OpenKAT 1.18 - Sneeuwkat

This release adds report scheduling, which implements periodic report generation: by adding an interval to a report, it will automatically update with the latest information. With our new Dashboarding feature, these reports can be added to custom dashboards.

Dashboard and Report data also have historical versions available. Future versions of the user interface will include moving back and forth in time and comparing these historic versions, highlighting changes and trends. We also included a new HTTP export boefje that you can use to export all objects in the graph to an external API either on an interval (e.g. every hour), or when the OOI is either created or changed, using our new Run-On functionality. This can be used to alert that findings have been created or their score has been updated. There’s also a new S3 backend for Bytes and various new boefjes, normalizers and fixes to bits from our growing community. Thanks! Docs on how to setup S3 for your (new) install can be found here: https://docs.openkat.nl/installation-and-deployment/s3-buckets.html

The language Tamil has been added via the hard work of a community volunteer. Since we have not yet tested it ourselves, it’s currently only available if you add it to the languages list manually. If you want to add a language to OpenKAT, or just help translate smaller parts, please take a look at our weblate: https://hosted.weblate.org/projects/openkat/nl-kat-coordination/ any help is much appreciated!

The Keiko module (formerly used to generate reports Via LaTeX) has been removed as we are now using full html reports that can also be exported as PDF.

In total 30 contributors have made 267 commits to Main in which 1,332 Files where changed.

New Features and Bug fixes

Documentation

Dependency Updates

Upgrading

Keiko has been removed. You should uninstall / remove the Keiko package or container.

Containers

When using docker with docker compose, you need to remove keiko from the docker-compose.yml file. You can then use –remove-orphan option to tell docker compose to remove containers that are no longer in the compose file:

docker compose up -d --remove-orphans

Debian packages

If you are using the Scripts we provide to install/upgrade OpenKAT you need to get the latest version that includes the kat-rocky-worker service.

You should also remove the kat-keiko package:

apt purge kat-keiko

Note that if you use the openkat-update.sh script to update to a newer 1.18 version (for example from 1.18.0rc1 to 1.18.0), then the kat-keiko will be installed again because the script will update or install all the packages. In 1.19 this won’t happen because the kat-keiko package will not exist anymore.

Deleting Outdated Reports

It was possible to test the new reporting feature in v1.17.0. However, v1.18.0 introduces newer versions of the Reports that are not compatible with these older models. If you have tested this feature in v1.17.0 or get a 500 error on the report (history) page, please use the database CLI tool to purge the old reporting data.

Script For The Containers

The following script should perform this operation. Check the –url (the XTDB_URI environment variable in a usual setup) and –node flags (the organisation code) and run the following command.

docker compose run --rm octopoes_api tools/xtdb-cli.py --url http://crux:3000 --node test evict-all-reports

Note that you should repeat the process for all organizations that face this issue. To find all your organisation codes, consider checking in Rocky or calling the KATalogus API:

docker compose run --rm -e DATABASE_MIGRATION=0 octopoes_api bash -c 'curl http://katalogus:8000/v1/organisations'
# To print only the organisation ids:
docker compose run --rm -e DATABASE_MIGRATION=0 octopoes_api bash -c 'curl -s http://katalogus:8000/v1/organisations | python -c "import sys, json; print(list(json.loads(sys.stdin.read()).keys()))"'

Script Tor The Debian Packages

The same holds for the debian packages, but invoking the script means having to set up the environment first. Check the –url (the XTDB_URI environment variable in a usual setup) and –node flags (the organisation code) and run the following command.

source /opt/venvs/kat-octopoes/bin/activate
export $(cat /usr/lib/kat/octopoes.defaults | grep -v "#") && export $(cat /etc/kat/octopoes.conf | grep -v "#")
cd /opt/venvs/kat-octopoes/lib/python3.11/site-packages
/opt/venvs/kat-octopoes/bin/python tools/xtdb-cli.py --url $XTDB_URI --node test evict-all-reports

To find all your organisation codes to apply this to multiple organisations, consider checking your organisations in Rocky or call the KATalogus API:

curl $KATALOGUS_API/v1/organisations
# To print only the organisation ids:
curl -s $KATALOGUS_API/v1/organisations | /opt/venvs/kat-octopoes/bin/python -c "import sys, json; print(list(json.loads(sys.stdin.read()).keys()))"

Full Changelog

The full changelog can be found on Github.