Aller au contenu

Infrastructure

This section gives you a complete overview of all the different components of Assemblyline, so you have a better idea of what they are used for.

Dependencies

The components listed here are external dependencies used by Assemblyline:

Dependency Description
Docker Docker is now at the heart of Assemblyline because all Assemblyline's components are now running as Docker containers. https://www.docker.com/
Kubernetes For multi-computer installations, Assemblyline uses Kubernetes to deploy the different Docker containers and keep them healthy. https://kubernetes.io/
Helm Helm is used to easily deploy and maintain our Kubernetes instance. https://helm.sh/
Elastic Stack Assemblyline uses the full Elastic stack to store results, logs, metrics, and APMs. It consists of the following components: https://www.elastic.co/elastic-stack
Elasticsearch Elasticsearch is used for storing results, logs, and metrics of the system. It also provides search capability to Assemblyline.
Kibana (optional) Provides dashboards to monitor your Assemblyline cluster.
APM (optional) Gather Application Performance Metrics so that we can pinpoint potential performance issues with the system and fix them.
Filebeat (optional) Gather all the logs for the different components into Elasticsearch to be displayed in Kibana.
Metricbeat (optional) Gather metrics for the different hosts where the Docker containers are run.
Redis We are using Redis for the queueing system, for messaging between the components, and as a remote data structure to keep multiple instances of a given component working in sync. https://redis.io/
Nginx Nginx is used by Assemblyline as a proxy to give the user access to the different user-facing components: UI, API, Socket Server, Kibana. https://www.nginx.com/
Minio For our default file storage, we use Minio which perfectly replicates the Amazon S3 API and is built to work with Kubernetes. https://min.io/

Core Components

The components listed here are Assemblyline-made processes that perform various tasks in the system:

Core components Description Link
Alerter Create alerts for the different submissions in the system. Source code
Dispatcher Route the files in the system while a submission is taking place. Make sure all files during a submission are completed by all required services. Source code
Expiry Delete submissions and their results when their TTL (Time-to-live) expires. Source code
Frontend Provides the user interface to interact with Assemblyline. Source code
Ingester Move ingested files from the priority queues to the processing queues. Source code
Metrics Aggregator Aggregate metrics of the different components in the system to save them into an ELK (Elasticsearch-Logstash-Kibana) stack. Source code
Metrics Heartbeat Provide live metrics in the system for the dashboard. Source code
Scaler Spin up and down services in the system depending on the load. Source code
Statistics Aggregator Generate daily statistics about signatures and heuristics. Source code
Updater Make sure the different services get their latest update files. Source code
Workflow Run the different workflows in the system and apply their labels, priority, and status. Source code
Service Server Provides an API for services to get tasks and post their results. Source code
UI / Socket Server Provides the APIs and a socket.io interface to interact with Assemblyline. Source code

Service interfaces

The interfaces listed here are used by Assemblyline's services to process files, generate results, and communicate back to the Service Server:

Service Interface Description Link
Python 2 Compatibility Library A library that gives services Python 2 compatibility. Source code
Result Class used by a service to generate results in the system. Source code
Service Base Base Assemblyline service class. Source code
Service Request Class that defines a request to scan a file for a given service. Source code
Task Handler A Python wrapper that communicates with the service server to get a task, download files, and publish results. It communicates with the service via named pipes so that the service can execute the received tasks. Source code

UI Plugins

The API may be extended via plugins. Plugins may be developed externally to Assemblyline to provide additional functionality. They must conform to the defined interface for their function. Currently, the only type of plugin defined is for information enrichment from systems external to Assemblyline.

Plugin Interface Description Link
External system lookups These plugins enable the querying of Assmebyline data in an external system and brining back the results. Source code