Infrastructure¶
This section gives you a complete overview of all the different components of Assemblyline, so you have a better idea of what they are used for.
Dependencies¶
The components listed here are external dependencies used by Assemblyline:
| Dependency | Description | |
|---|---|---|
| Docker | Docker is now at the heart of Assemblyline because all Assemblyline's components are now running as Docker containers. | https://www.docker.com/ |
| Kubernetes | For multi-computer installations, Assemblyline uses Kubernetes to deploy the different Docker containers and keep them healthy. | https://kubernetes.io/ |
| Helm | Helm is used to easily deploy and maintain our Kubernetes instance. | https://helm.sh/ |
| Elastic Stack | Assemblyline uses the full Elastic stack to store results, logs, metrics, and APMs. It consists of the following components: | https://www.elastic.co/elastic-stack |
| Elasticsearch | Elasticsearch is used for storing results, logs, and metrics of the system. It also provides search capability to Assemblyline. | |
| Kibana (optional) | Provides dashboards to monitor your Assemblyline cluster. | |
| APM (optional) | Gather Application Performance Metrics so that we can pinpoint potential performance issues with the system and fix them. | |
| Filebeat (optional) | Gather all the logs for the different components into Elasticsearch to be displayed in Kibana. | |
| Metricbeat (optional) | Gather metrics for the different hosts where the Docker containers are run. | |
| Redis | We are using Redis for the queueing system, for messaging between the components, and as a remote data structure to keep multiple instances of a given component working in sync. | https://redis.io/ |
| Nginx | Nginx is used by Assemblyline as a proxy to give the user access to the different user-facing components: UI, API, Socket Server, Kibana. | https://www.nginx.com/ |
| Minio | For our default file storage, we use Minio which perfectly replicates the Amazon S3 API and is built to work with Kubernetes. | https://min.io/ |
Core Components¶
The components listed here are Assemblyline-made processes that perform various tasks in the system:
| Core components | Description | Link |
|---|---|---|
| Alerter | Create alerts for the different submissions in the system. | Source code |
| Dispatcher | Route the files in the system while a submission is taking place. Make sure all files during a submission are completed by all required services. | Source code |
| Expiry | Delete submissions and their results when their TTL (Time-to-live) expires. | Source code |
| Frontend | Provides the user interface to interact with Assemblyline. | Source code |
| Ingester | Move ingested files from the priority queues to the processing queues. | Source code |
| Metrics Aggregator | Aggregate metrics of the different components in the system to save them into an ELK (Elasticsearch-Logstash-Kibana) stack. | Source code |
| Metrics Heartbeat | Provide live metrics in the system for the dashboard. | Source code |
| Scaler | Spin up and down services in the system depending on the load. | Source code |
| Statistics Aggregator | Generate daily statistics about signatures and heuristics. | Source code |
| Updater | Make sure the different services get their latest update files. | Source code |
| Workflow | Run the different workflows in the system and apply their labels, priority, and status. | Source code |
| Service Server | Provides an API for services to get tasks and post their results. | Source code |
| UI / Socket Server | Provides the APIs and a socket.io interface to interact with Assemblyline. | Source code |
Service interfaces¶
The interfaces listed here are used by Assemblyline's services to process files, generate results, and communicate back to the Service Server:
| Service Interface | Description | Link |
|---|---|---|
| Python 2 Compatibility Library | A library that gives services Python 2 compatibility. | Source code |
| Result | Class used by a service to generate results in the system. | Source code |
| Service Base | Base Assemblyline service class. | Source code |
| Service Request | Class that defines a request to scan a file for a given service. | Source code |
| Task Handler | A Python wrapper that communicates with the service server to get a task, download files, and publish results. It communicates with the service via named pipes so that the service can execute the received tasks. | Source code |
UI Plugins¶
The API may be extended via plugins. Plugins may be developed externally to Assemblyline to provide additional functionality. They must conform to the defined interface for their function. Currently, the only type of plugin defined is for information enrichment from systems external to Assemblyline.
| Plugin Interface | Description | Link |
|---|---|---|
| External system lookups | These plugins enable the querying of Assmebyline data in an external system and brining back the results. | Source code |