Infrastructure

This section gives you a complete overview of all the different components of Assemblyline, so you have a better idea of what they are used for.

Dependencies

The components listed here are external dependencies used by Assemblyline:

Dependency	Description
Docker	Docker is now at the heart of Assemblyline because all Assemblyline's components are now running as Docker containers.	https://www.docker.com/
Kubernetes	For multi-computer installations, Assemblyline uses Kubernetes to deploy the different Docker containers and keep them healthy.	https://kubernetes.io/
Helm	Helm is used to easily deploy and maintain our Kubernetes instance.	https://helm.sh/
Elastic Stack	Assemblyline uses the full Elastic stack to store results, logs, metrics, and APMs. It consists of the following components:	https://www.elastic.co/elastic-stack
	Elasticsearch	Elasticsearch is used for storing results, logs, and metrics of the system. It also provides search capability to Assemblyline.
	Kibana (optional)	Provides dashboards to monitor your Assemblyline cluster.
	APM (optional)	Gather Application Performance Metrics so that we can pinpoint potential performance issues with the system and fix them.
	Filebeat (optional)	Gather all the logs for the different components into Elasticsearch to be displayed in Kibana.
	Metricbeat (optional)	Gather metrics for the different hosts where the Docker containers are run.
Redis	We are using Redis for the queueing system, for messaging between the components, and as a remote data structure to keep multiple instances of a given component working in sync.	https://redis.io/
Nginx	Nginx is used by Assemblyline as a proxy to give the user access to the different user-facing components: UI, API, Socket Server, Kibana.	https://www.nginx.com/
Minio	For our default file storage, we use Minio which perfectly replicates the Amazon S3 API and is built to work with Kubernetes.	https://min.io/

Core Components

The components listed here are Assemblyline-made processes that perform various tasks in the system:

Core components	Description	Link
Alerter	Create alerts for the different submissions in the system.	Source code
Dispatcher	Route the files in the system while a submission is taking place. Make sure all files during a submission are completed by all required services.	Source code
Expiry	Delete submissions and their results when their TTL (Time-to-live) expires.	Source code
Frontend	Provides the user interface to interact with Assemblyline.	Source code
Ingester	Move ingested files from the priority queues to the processing queues.	Source code
Metrics Aggregator	Aggregate metrics of the different components in the system to save them into an ELK (Elasticsearch-Logstash-Kibana) stack.	Source code
Metrics Heartbeat	Provide live metrics in the system for the dashboard.	Source code
Scaler	Spin up and down services in the system depending on the load.	Source code
Statistics Aggregator	Generate daily statistics about signatures and heuristics.	Source code
Updater	Make sure the different services get their latest update files.	Source code
Workflow	Run the different workflows in the system and apply their labels, priority, and status.	Source code
Service Server	Provides an API for services to get tasks and post their results.	Source code
UI / Socket Server	Provides the APIs and a socket.io interface to interact with Assemblyline.	Source code

Service interfaces

The interfaces listed here are used by Assemblyline's services to process files, generate results, and communicate back to the Service Server:

Service Interface	Description	Link
Python 2 Compatibility Library	A library that gives services Python 2 compatibility.	Source code
Result	Class used by a service to generate results in the system.	Source code
Service Base	Base Assemblyline service class.	Source code
Service Request	Class that defines a request to scan a file for a given service.	Source code
Task Handler	A Python wrapper that communicates with the service server to get a task, download files, and publish results. It communicates with the service via named pipes so that the service can execute the received tasks.	Source code

UI Plugins

The API may be extended via plugins. Plugins may be developed externally to Assemblyline to provide additional functionality. They must conform to the defined interface for their function. Currently, the only type of plugin defined is for information enrichment from systems external to Assemblyline.

Plugin Interface	Description	Link
External system lookups	These plugins enable the querying of Assmebyline data in an external system and brining back the results.	Source code