Nov 28, 2019

Platform Architecture

Welcome to the ingenious world of DEX8 Platform Architecture. As you embark on this journey, you'll uncover a meticulously crafted orchestration of components that continuously interact to unleash the power of data extraction and automation.

Terminology

• skript file - a text file containing JavaScript code, with an extension ending in .js
• skript - the worker's task, consisting of one or many skript files, the most basic being a main.js file.
• worker - executes a skript, storing extracted data to a database and transmitting real-time echo messages to the Web Panel.
• storage - typically an SQL or NoSQL database where extracted data is stored.
• exporter - exports data from the database into formats like CSV, XLS, or PDF.
• Joint API - enables integration between external applications or services and the DEX8 platform.
• scheduler - schedules skript execution using cron jobs.

Platform Components

• Skript Files: JavaScript files written for the Node.js environment.
• Skript: A set of instructions defining what a worker should do.
• Worker: A Node.js instance executing skripts on a Linux or Windows server.
• Web Panel: A browser-based GUI for writing skripts, managing workers, and exporting data.
• API: Connects the Web Panel and workers, handling communication and monitoring.
• Database: A NoSQL database storing extracted data in real time.
• Exporter: Converts stored data into various formats for external use.
• Scheduler: Automates skript execution at predefined intervals.

How does the DEX8 platform work?

When a user clicks the START button on the Web Panel, a skript is sent to workers via API. Workers then extract data from web pages, store it in a database, and send real-time logs to the Web Panel.

Workers retrieve and process data according to the skript.

Extracted data is stored in the database while real-time messages are sent to the Web Panel.