Bytes

Bytes is a service that provides an API for reading and writing metadata of jobs and job outputs (raw files). It can also encrypt the raw data, and hash it to proof that the data was seen before a point in time (see below).

Installation

There are two ways to setup the API.

With Docker

Bytes can be fired up from the root directory of KAT using docker-compose (check out the README over there!).

To run Bytes as a standalone container, spin up a Postgresql database (e.g. using Docker), create the database bytes and run

$ docker build . -t bytes

# Without an env-file
$ export BYTES_PASSWORD=$(openssl rand -hex 20) \
    && export BYTES_SECRET=$(openssl rand -hex 20) \
    && export BYTES_DB_URI=postgresql://USER:PWD@bytes-db:5432/bytes  # change accordingly!
$ docker run --rm -p 8002:8002 -e BYTES_USERNAME=bytes -e BYTES_PASSWORD -e BYTES_SECRET -e BYTES_DB_URI bytes


# With an env-file
$ docker run --rm -p 8002:8000 --env-file=/path/to/env bytes  # change accordingly!

Without Docker

To create and start a Python virtual environment, run

$ python -m venv $PWD/.venv
$ source .venv/bin/activate

To install the dependencies, assuming you are in the virtual environment, run

$ pip install -r requirements-dev.txt

Bytes depends on a Postgresql database that is configurable by the BYTES_DB_URI environment variable. See above for a minimal set of environment variables to start Bytes and

To start the API run

$ uvicorn bytes.api:app --host 127.0.0.1 --port 8002 --reload --reload-dir /app/bytes/bytes

See http://localhost:8002/docs for the OpenAPI documentation.

Hashing and Encryption

Every raw file is hashed with the current ended_at of the boefje_meta, which functions as a ‘proof’ of it being uploaded at that time. These proofs can be uploaded externally (a 3rd party) such that we can verify that this data was saved in the past.

Current implementations are

  • BYTES_EXT_HASH_REPOSITORY="IN_MEMORY" (just a stub)

  • BYTES_EXT_HASH_REPOSITORY="PASTEBIN" (Needs pastebin API development key)

  • BYTES_EXT_HASH_REPOSITORY="RFC3161"

For the RFC3161 implementation, see https://www.ietf.org/rfc/rfc3161.txt and https://github.com/trbs/rfc3161ng as a reference. To use this implementation, set your environment to

  • BYTES_EXT_HASH_REPOSITORY=RFC3161

  • BYTES_RFC3161_PROVIDER="https://freetsa.org/tsr" (example)

  • BYTES_RFC3161_CERT_FILE="bytes/timestamping/certificates/freetsa.crt" (example)

Adding a new implementation means implementing the bytes.repositories.hash_repository::HashRepository interface. Bind your new implementation in bytes.timestamping.provider::create_hash_repository.

The secure-hashing-algorithm can be specified with an env var: BYTES_HASHING_ALGORITHM="SHA512".

BYTES_HASHING_ALGORITHM="SHA512"
BYTES_EXT_HASH_REPOSITORY="IN_MEMORY"
BYTES_PASTEBIN_API_DEV_KEY=""

Files in bytes can be saved encrypted to disk, the implementation can be set using an env-var, BYTES_ENCRYPTION_MIDDLEWARE. The options are:

  • "IDENTITY"

  • "NACL_SEALBOX"

The "NACL_SEALBOX" option requires the BYTES_PRIVATE_KEY_B64 and BYTES_PUBLIC_KEY_B64 env vars.

Observability

Bytes exposes a /metrics endpoint for basic application level observability, such as the amount of organizations and the amount of raw files per organization. Another important component to monitor is the disk usage of Bytes. It is recommended to install node exporter to keep track of this.

Design

We now include two levels of design, according to the C4 model.

Design: C2 Container level

The overall view of the code is as follows.

graph User((User)) Rocky["Rocky<br/><i>Django App</i>"] Bytes{"Bytes<br/><i>FastAPI App"} RabbitMQ[["RabbitMQ<br/><i>Message Broker"]] Scheduler["Scheduler<br/><i>Software System"] Boefjes["Boefjes<br/><i>Python App"] Boefjes -- GET/POST Raw/Meta --> Bytes User -- Interacts with --> Rocky Rocky -- GET/POST Raw/Meta --> Bytes Bytes -- "publish(RawFileReceived)" --> RabbitMQ Scheduler --"subscribe(RawFileReceived)"--> RabbitMQ Scheduler --"GET BoefjeMeta"--> Bytes

Design: C3 Component level

The overall view of the code is as follows.

graph LR User -- BoefjeMeta --> APIR1 User -- NormalizerMeta --> APIR2 User -- RawFile --> APIR3 User[User] APIR1 -- save BoefjeMeta --> MR APIR2 -- save NormalizerMeta --> MR APIR3 -- save RawFile --> MR subgraph API["Bytes API"] APIR1[API Route] APIR2[API Route] APIR3[API Route] end subgraph Bytes["Bytes Domain"] APIR3 -- "publish(RawFileReceived)" --> EM[EventManager] MR[Meta Repository] -- Raw --> H[Hasher] -- Hash --> MR[Meta Repository] MR[Meta Repository] -- save Hash --> HR[Hash Repository] MR[Meta Repository] -- save RawFile --> R[Raw Repository] R[Raw Repository] -- RawFile --> F[FileMiddleware] end F[FileMiddleware] -- Encrypted Data --> Disk[[Disk]] -- Encrypted Data --> F[FileMiddleware] HR[Hash Repository] -- Hash --> T[[Third Party]] MR[Meta Repository] -- BoefjeMeta/NormalizerMeta --> RDB[(Psql)] EM[EventManager] -- "{'event_id': 123}" --> RabbitMQ[[RabbitMQ]]

This diagram roughly covers the C4 level as well, as this is a small service that can be regarded as one component.

Development

The Makefile provides useful targets to use during development. To see the options run

$ make help

Code style and tests

All the code style and linting checks are done by running

$ make check

The unit and integration tests targets are utest and itest respectively. To run all test, run

$ make test

To make sure all github actions (checks and tests) pass, run

$ make done

Ideally, you run this before each commit. Passing all the checks and tests in this target should ensure the github actions pass.

Migrations

To make a new migration file and run the migration, run

$ make migrations m='Some migration message'
$ make migrate

Export SQL migrations

To export raw SQL from the SQLAlchemy migration files, run the following target (for the diff between 0003 and 0004):

$ make sql rev1=0003 rev2=0004 > sql_migrations/0004_change_x_to_y_add_column_z.sql

Production

Performance tuning

Bytes caches some metrics for performance, but the default is not to cache these queries. It is recommended to tune the BYTES_METRICS_TTL_SECONDS variable to on the amount of calls to the /metrics endpoint. As a guideline, add at least 10 seconds to the cache for every million of raw files in the database.