Production: Container deployment
OpenKAT can be deployed using containers. We aim to support both simple docker / docker-compose setups and container orchestration systems like Kubernetes and Nomad.
There is a docker-compose.release-example.yml in the root directory that can be used as an example how to deploy using docker-compose.
Container images
The container images can be found here:
https://github.com/minvws/nl-kat-boefjes/pkgs/container/nl-kat-boefjes
https://github.com/minvws/nl-kat-bytes/pkgs/container/nl-kat-bytes
https://github.com/minvws/nl-kat-mula/pkgs/container/nl-kat-mula
https://github.com/minvws/nl-kat-octopoes/pkgs/container/nl-kat-octopoes
https://github.com/minvws/nl-kat-rocky/pkgs/container/nl-kat-rocky
https://github.com/minvws/nl-kat-keiko/pkgs/container/nl-kat-keiko
Setup
To set up an installation with pre-built containers, you can pull the repository using:
git clone https://github.com/minvws/nl-kat-coordination.git
If this is your first install, and you do not have an .env file yet, you can create an .env file using the following command:
make env
This will create an .env file with the default values. You can edit this file to change the default values. Now you can pull and start the containers using the following command:
docker-compose -f docker-compose.release-example.yml up -d
The container image run the necessary database migration commands in the entrypoint if DATABASE_MIGRATION is set. You manually need to run setup commands in the katalogus and rocky containers to initialize everything. In the katalogus container we need to create an organisation, we can do this by running the following in the katalogus container:
python3 -m boefjes.seed
With docker-compose you would run this as:
docker-compose -f docker-compose.release-example.yml exec katalogus python3 -m boefjes.seed
In the rocky container we first need to import the OOI database seed:
python3 manage.py loaddata OOI_database_seed.json
With docker-compose you would run this as:
docker-compose -f docker-compose.release-example.yml exec rocky python3 manage.py loaddata OOI_database_seed.json
Next we need to create the superuser, this will prompt for the e-mail address and password:
python3 manage.py createsuperuser
With docker-compose you would run this as:
docker-compose -f docker-compose.release-example.yml exec rocky python3 manage.py createsuperuser
We also need to create an organisation, this command will create a development organisation:
python3 manage.py setup_dev_account
With docker-compose you would run this as:
docker-compose -f docker-compose.release-example.yml exec rocky python3 manage.py setup_dev_account
Container commands
We have two container images that are used to run multiple containers. What the container runs is be specified by overriding the CMD of the container.
Container image |
CMD |
Description |
|---|---|---|
boefjes |
boefje |
Boefjes runtime |
boefjes |
normalizer |
Normalizers runtime |
boefjes |
katalogus |
Katalogus API |
octopoes |
web |
Octopoes API |
octopoes |
worker-beat |
Celery worker running beat. There must only be exactly one container of this type |
octopoes |
worker |
Celery worker. Use this if you need to more than one work container for scaling |
Env variables
Each container needs to be configured using a set of environment variables
Boefjes / Katalogus
Environment variable |
Required |
Default Value |
Format |
Description |
|---|---|---|---|---|
|
no |
10 |
Number of worker processes to start |
|
|
no |
boefjes |
Queue name for boefjes |
|
|
no |
normalizers |
Queue name for normalizers |
|
|
yes |
amqp://user:pass@host:5672/vhost |
RabbitMQ used by celery, should be the same as Mula |
|
|
yes |
http://bytes:8000 |
URI for the Bytes API |
|
|
yes |
Username for Bytes API |
||
|
yes |
Password for Bytes API |
||
|
yes |
http://katalogus:8000 |
URI for the Katalogus API |
|
|
yes |
http://octopoes_api:8000 |
URI for the Octopoes API |
|
|
yes |
postgresql://user:paswd@host:5432/db |
URI for the Postgresql DB |
|
|
no |
A token needed by WP Scan boefje |
||
|
no |
IDENTITY |
Encryption to use for the katalogus settings: IDENTITY (no encryption) or NACL_SEALBOX |
|
|
no |
KATalogus NaCl Sealbox base-64 private key string |
||
|
no |
KATalogus NaCl Sealbox base-64 public key string |
||
|
no |
|||
|
no |
|||
|
no |
|||
|
no |
|||
|
no |
8.8.8.8 |
||
|
no |
|||
|
no |
|||
|
no |
false |
Container entrypoint will run database migrations if set to “true” |
See also https://github.com/minvws/nl-kat-boefjes#environment-variables
Bytes
Environment variable |
Required |
Default Value |
Format |
Description |
|---|---|---|---|---|
|
yes |
Secret used for JWT |
||
|
yes |
Username for Bytes API |
||
|
yes |
Password for Bytes API |
||
|
no |
RabbitMQ queue to send events to |
||
|
yes |
postgresql://user:paswd@host:5432/db |
URI for the Postgresql DB |
|
|
yes |
Directory to store files |
||
|
yes |
|
Encryption to use: IDENTITY (no encryption) or NACL_SEALBOX |
|
|
no |
false |
Container entrypoint will run database migrations if set to “true” |
See also https://github.com/minvws/nl-kat-bytes#configuration
Octopoes
Environment variable |
Required |
Default Value |
Format |
Description |
|---|---|---|---|---|
|
yes |
http://crux:3000 |
XTDB uri |
|
|
no |
crux |
crux, xtdb or xtdb-multinode |
|
|
yes |
RabbitMQ queue |
||
|
yes |
http://katalogus:8000 |
URI for the Katalogus API |
See also https://github.com/minvws/nl-kat-octopoes#environment-variables
Mula
Environment variable |
Required |
Default Value |
Format |
Description |
|---|---|---|---|---|
|
no |
False |
Set to True to enable queueing of boefjes |
|
|
no |
True |
Set to True to enable queueing of normalizers |
|
|
yes |
amqp://user:pass@host:5672/vhost |
RabbitMQ instance used by scheduler, can be the same as celery |
|
|
yes |
postgresql://user:paswd@host:5432/db |
URI for scheduler DB |
|
|
yes |
http://bytes:8000 |
URI for the Bytes API |
|
|
yes |
Username for Bytes API |
||
|
yes |
Password for Bytes API |
||
|
yes |
http://katalogus:8000 |
URI for the Katalogus API |
|
|
yes |
http://octopoes_api:8000 |
URI for the Octopoes API |
|
|
no |
false |
Container entrypoint will run database migrations if set to “true” |
See also https://github.com/minvws/nl-kat-mula/blob/main/docs/configuration.md
Rocky
Environment variable |
Required |
Default Value |
Format |
Description |
|---|---|---|---|---|
|
yes |
Postgres host |
||
|
yes |
Postgres port |
||
|
yes |
Postgres database database |
||
|
yes |
Postgres username |
||
|
yes |
Postgres password |
||
|
yes |
String |
Key of at least 50 characters, see https://docs.djangoproject.com/en/4.1/ref/settings/#secret-key |
|
|
no |
boefjes |
Queue name for boefjes |
|
|
no |
normalizers |
Queue name for normalizers |
|
|
yes |
amqp://user:pass@host:5672/vhost |
RabbitMQ used by celery, should be the same as Mula |
|
|
yes |
http://bytes:8000 |
URI for the Bytes API |
|
|
yes |
Username for Bytes API |
||
|
yes |
Password for Bytes API |
||
|
yes |
http://katalogus:8000 |
URI for the Katalogus API |
|
|
yes |
http://octopoes_api:8000 |
URI for the Octopoes API |
|
|
yes |
http://scheduler:8000 |
URI for the scheduler API |
|
|
no |
Hostname of mail server to use to send e-mails |
||
|
no |
25 |
Mail server port |
|
|
no |
Username to use to connect to mail server |
||
|
no |
Password to use to connect to mail server |
||
|
no |
https://docs.djangoproject.com/en/4.1/ref/settings/#default-from-email |
||
|
no |
https://docs.djangoproject.com/en/4.1/ref/settings/#server-email |
||
|
no |
https://docs.djangoproject.com/en/4.1/ref/settings/#email-use-tls |
||
|
no |
https://docs.djangoproject.com/en/4.1/ref/settings/#email-use-ssl |
||
|
no |
false |
Container entrypoint will run database migrations if set to “true” |
Upgrading
When deploying new container images the database migrations are automatically run in the entrypoint. The OOI_database_seed.json file needs to be loaded manually using the following command:
python3 manage.py loaddata OOI_database_seed.json
With docker-compose you would run this as:
docker-compose -f docker-compose.release-example.yml exec rocky python3 manage.py loaddata OOI_database_seed.json