Developer documentation

Contains documentation for developers and contributors.

Contents

• Contributing
  • Contributing
    • Introduction
  • UI/UX design documentation
    • Figma
      • Design process
      • Fundamentals
      • Components
      • Design workfile
      • For review
      • Approved for implementation
  • Project Guidelines
    • Project management
      • Feature Milestones
      • Bugs and Feature Requests
      • Pull Requests
      • In-depth content discussions
    • Development
      • Code
      • Tools
      • Signed commits
      • Type Hinting
      • Testing
      • Development Environment
      • Merge Strategy
      • Line ends
      • Technical diagrams
      • Dependency management
    • OpenKAT background and concepts
      • Indemnification statements
      • Safe viewing of boefjes-data (dutch)
      • OpenKAT as educational tool (dutch)
    • Contributions
      • Contribute to Codebase
      • Contribute Documentation
      • Contribute Translations
      • Contributor Social Contract
      • Code of Conduct
      • Security
    • Feature flow
      • Getting features in the main branch
      • Releasing features
      • Tips and tricks for pull request QA testing
    • Security
      • Security in the pipeline
  • GitHub Templates
    • Bug Report Template
    • Feature Request Template
    • Pull Request template for authors
    • Code review checklist for PR’s
    • QA checklist for PR’s
• Quick start
  • Installation
• Basic principles
  • Basics of OpenKAT
    • General notes
    • Basic concepts
      • Objects, the data model and recursion
      • Indemnification
      • Users and organizations
    • System design
      • Collection: Boefjes and Whiskers
      • Storage: Bytes and Octopoes
      • Analysis: Bits
      • Reports
  • Modules
    • Rocky - frontend
    • Mula - scheduler
    • Octopoes - datamodel
    • Bytes - raw data storage
    • Boefjes and whiskers - scanners and normalizers
    • Manon - styling
  • Boefjes
  • Origin types
  • Bits
  • Normalizers
    • Burp Suite
  • Questions & Configs
    • Disallowed CSP hostnames
    • OOIs in headers
    • Port mapping
  • Plugins for OpenKAT: boefjes, whiskers and bits
    • What types of plugins are available?
    • How does it work?
    • Where to start?
    • Existing boefjes
      • Object-types, classes and objects.
      • Example: the boefje for shodan
      • Normalizers
    • Adding object-types
    • Bits: businessrules
    • Configurable bits
      • Question object
      • My first question object
      • What happens in the background?
      • Advantages and outlook
    • Adding Boefjes
  • Trusted timestamps in OpenKAT
    • About the protocol
    • Available timestamp servers
    • How to verify a timestamp?
      • Download the raw data
      • Verify the hash
      • Verify the timestamp
      • Automation of the verification process
• Rocky
  • Installation
    • Containerized
    • Local
      • Django App
      • Frontend
      • TL;DR
  • Development
    • Testing
      • Tip
  • Design
    • Fonts license
      • Fredoka
      • Open Sans
      • Tabler icons
  • Technical Design
    • Running a boefje
    • Rocky View Structure
      • Overview of child Views of the OrganizationViews
      • Exhaustive overview of OctopoesViews
      • KATalogus Views
• Scheduler
  • Purpose
  • Architecture
    • Stack, packages and libraries
    • External services
    • Project structure
  • Running / Developing
    • Prerequisites
    • Running
    • Migrations
  • Testing
  • Scripts
  • load.py
  • benchmark.py
• Boefjes
  • Prerequisites
  • KATalogus
    • Organisations
    • Plugins
    • Settings
  • Environment variables
  • Technical Design
    • Boefje and Normalizer Workers
      • Parallel Workers
      • Sending a SIGKILL to a worker process
    • Running as a Docker container
    • Running the worker directly
    • Example job
    • Manually running a boefje or normalizer
    • Boefje and normalizer structure
      • Example
    • Tests
• Design considerations for new boefjes runner
  • Images
    • Distribution
    • Metadata
  • I/O
    • Input
    • Output
  • Logging
  • Runtimes
    • Docker
    • Kubernetes
    • Nomad
  • Building images with this spec from the current boefjes
    • Summary of decisions
  • Limitations
• Design considerations for new normalisers (whiskers) runner
  • Current situation
  • Requirements
  • Design
    • Runtime
      • MicroVM (Firecracker)
      • Containers (Docker)
      • Containers (hardened runtimes)
      • WebAssembly (Wasm)
      • Others
    • Distribution
    • Input/Output
    • Supervisor process
  • Discussion
  • Conclusions
• Bytes
  • Installation
    • With Docker
    • Without Docker
    • Hashing and Encryption
    • Observability
  • Design
    • Design: C2 Container level
    • Design: C3 Component level
  • Development
    • Code style and tests
    • Migrations
    • Export SQL migrations
  • Production
    • Performance tuning
• Octopoes
  • Instructions
    • Run Octopoes API
    • Run the event processor
  • Healthcheck
  • OOI
  • Origin
  • Origin through declaration
  • Origin through observation
  • Origin through inference
  • Graph mutations
  • Code Architecture
    • Sequence: save_origin
    • Sequence: process update ooi
  • XTDB
    • XTDB-cli tool
      • Examples
    • XTDB analyze bits tool
  • OOI Objects
  • Relationships
  • A few example records
  • OOI Reference
  • Octopoes API
    • OctopoesAPIConnector
      • Querying objects
  • Abstract classes / subclassing
  • Querying
  • Run bit manually
  • Tests
• Octopoes Models
  • octopoes.models.ooi.service
    • IPService
      • IPService.ip_port
      • IPService.object_type
      • IPService.service
      • IPService.format_reference_human_readable()
      • IPService.model_post_init()
    • Service
      • Service.name
      • Service.object_type
      • Service.format_reference_human_readable()
      • Service.model_post_init()
    • TLSCipher
      • TLSCipher.ip_service
      • TLSCipher.object_type
      • TLSCipher.suites
      • TLSCipher.format_reference_human_readable()
      • TLSCipher.model_post_init()
  • octopoes.models.ooi.findings
    • ADRFindingType
      • ADRFindingType.object_type
    • CAPECFindingType
      • CAPECFindingType.id
      • CAPECFindingType.object_type
    • CVEFindingType
      • CVEFindingType.id
      • CVEFindingType.object_type
    • CWEFindingType
      • CWEFindingType.id
      • CWEFindingType.object_type
    • Finding
      • Finding.description
      • Finding.finding_type
      • Finding.object_type
      • Finding.ooi
      • Finding.proof
      • Finding.reproduce
      • Finding.format_reference_human_readable()
      • Finding.model_post_init()
      • Finding.natural_key
    • FindingType
      • FindingType.description
      • FindingType.id
      • FindingType.impact
      • FindingType.name
      • FindingType.recommendation
      • FindingType.risk_score
      • FindingType.risk_severity
      • FindingType.source
      • FindingType.format_reference_human_readable()
      • FindingType.model_post_init()
    • KATFindingType
      • KATFindingType.object_type
    • MutedFinding
      • MutedFinding.finding
      • MutedFinding.object_type
      • MutedFinding.reason
      • MutedFinding.format_reference_human_readable()
      • MutedFinding.model_post_init()
    • RetireJSFindingType
      • RetireJSFindingType.object_type
    • RiskLevelSeverity
    • SnykFindingType
      • SnykFindingType.id
      • SnykFindingType.object_type
  • octopoes.models.ooi.email_security
    • DKIMExists
      • DKIMExists.hostname
      • DKIMExists.object_type
      • DKIMExists.format_reference_human_readable()
      • DKIMExists.model_post_init()
    • DKIMKey
      • DKIMKey.dkim_selector
      • DKIMKey.key
      • DKIMKey.object_type
      • DKIMKey.format_reference_human_readable()
      • DKIMKey.model_post_init()
    • DKIMSelector
      • DKIMSelector.hostname
      • DKIMSelector.object_type
      • DKIMSelector.selector
      • DKIMSelector.format_reference_human_readable()
      • DKIMSelector.model_post_init()
    • DMARCTXTRecord
      • DMARCTXTRecord.hostname
      • DMARCTXTRecord.object_type
      • DMARCTXTRecord.ttl
      • DMARCTXTRecord.value
      • DMARCTXTRecord.format_reference_human_readable()
      • DMARCTXTRecord.model_post_init()
    • DNSSPFMechanism
      • DNSSPFMechanism.mechanism
      • DNSSPFMechanism.spf_record
    • DNSSPFMechanismHostname
      • DNSSPFMechanismHostname.hostname
      • DNSSPFMechanismHostname.object_type
      • DNSSPFMechanismHostname.qualifier
      • DNSSPFMechanismHostname.format_reference_human_readable()
      • DNSSPFMechanismHostname.model_post_init()
    • DNSSPFMechanismIP
      • DNSSPFMechanismIP.ip
      • DNSSPFMechanismIP.object_type
      • DNSSPFMechanismIP.qualifier
      • DNSSPFMechanismIP.format_reference_human_readable()
      • DNSSPFMechanismIP.model_post_init()
    • DNSSPFMechanismNetBlock
      • DNSSPFMechanismNetBlock.netblock
      • DNSSPFMechanismNetBlock.object_type
      • DNSSPFMechanismNetBlock.qualifier
      • DNSSPFMechanismNetBlock.format_reference_human_readable()
      • DNSSPFMechanismNetBlock.model_post_init()
    • DNSSPFRecord
      • DNSSPFRecord.all
      • DNSSPFRecord.dns_txt_record
      • DNSSPFRecord.exp
      • DNSSPFRecord.object_type
      • DNSSPFRecord.ttl
      • DNSSPFRecord.value
      • DNSSPFRecord.format_reference_human_readable()
      • DNSSPFRecord.model_post_init()
      • DNSSPFRecord.natural_key
    • MechanismQualifier
  • octopoes.models.ooi.web
    • APIDesignRule
      • APIDesignRule.name
      • APIDesignRule.object_type
      • APIDesignRule.format_reference_human_readable()
      • APIDesignRule.model_post_init()
    • APIDesignRuleResult
      • APIDesignRuleResult.message
      • APIDesignRuleResult.object_type
      • APIDesignRuleResult.passed
      • APIDesignRuleResult.rest_api
      • APIDesignRuleResult.rule
      • APIDesignRuleResult.format_reference_human_readable()
      • APIDesignRuleResult.model_post_init()
    • HTTPHeader
      • HTTPHeader.key
      • HTTPHeader.object_type
      • HTTPHeader.resource
      • HTTPHeader.value
      • HTTPHeader.format_reference_human_readable()
      • HTTPHeader.model_post_init()
    • HTTPHeaderHostname
      • HTTPHeaderHostname.header
      • HTTPHeaderHostname.hostname
      • HTTPHeaderHostname.object_type
      • HTTPHeaderHostname.format_reference_human_readable()
      • HTTPHeaderHostname.model_post_init()
    • HTTPHeaderURL
      • HTTPHeaderURL.header
      • HTTPHeaderURL.object_type
      • HTTPHeaderURL.url
      • HTTPHeaderURL.format_reference_human_readable()
      • HTTPHeaderURL.model_post_init()
    • HTTPResource
      • HTTPResource.object_type
      • HTTPResource.redirects_to
      • HTTPResource.web_url
      • HTTPResource.website
      • HTTPResource.format_reference_human_readable()
      • HTTPResource.model_post_init()
    • HostnameHTTPURL
      • HostnameHTTPURL.netloc
      • HostnameHTTPURL.object_type
      • HostnameHTTPURL.format_reference_human_readable()
      • HostnameHTTPURL.model_post_init()
    • IPAddressHTTPURL
      • IPAddressHTTPURL.netloc
      • IPAddressHTTPURL.object_type
      • IPAddressHTTPURL.format_reference_human_readable()
      • IPAddressHTTPURL.model_post_init()
    • ImageMetadata
      • ImageMetadata.image_info
      • ImageMetadata.object_type
      • ImageMetadata.resource
      • ImageMetadata.format_reference_human_readable()
      • ImageMetadata.model_post_init()
    • RESTAPI
      • RESTAPI.api_url
      • RESTAPI.object_type
      • RESTAPI.format_reference_human_readable()
      • RESTAPI.model_post_init()
    • SecurityTXT
      • SecurityTXT.object_type
      • SecurityTXT.redirects_to
      • SecurityTXT.security_txt
      • SecurityTXT.url
      • SecurityTXT.website
      • SecurityTXT.model_post_init()
    • URL
      • URL.network
      • URL.object_type
      • URL.raw
      • URL.web_url
      • URL.format_reference_human_readable()
      • URL.model_post_init()
    • WebScheme
    • WebURL
      • WebURL.network
      • WebURL.path
      • WebURL.port
      • WebURL.scheme
    • Website
      • Website.certificate
      • Website.hostname
      • Website.ip_service
      • Website.object_type
      • Website.format_reference_human_readable()
      • Website.model_post_init()
  • octopoes.models.ooi.config
    • Config
      • Config.bit_id
      • Config.config
      • Config.object_type
      • Config.ooi
      • Config.format_reference_human_readable()
      • Config.model_post_init()
  • octopoes.models.ooi.certificate
    • AlgorithmType
    • SubjectAlternativeName
      • SubjectAlternativeName.certificate
      • SubjectAlternativeName.model_post_init()
    • SubjectAlternativeNameHostname
      • SubjectAlternativeNameHostname.hostname
      • SubjectAlternativeNameHostname.object_type
      • SubjectAlternativeNameHostname.format_reference_human_readable()
      • SubjectAlternativeNameHostname.model_post_init()
    • SubjectAlternativeNameIP
      • SubjectAlternativeNameIP.address
      • SubjectAlternativeNameIP.object_type
      • SubjectAlternativeNameIP.format_reference_human_readable()
      • SubjectAlternativeNameIP.model_post_init()
    • SubjectAlternativeNameQualifier
      • SubjectAlternativeNameQualifier.name
      • SubjectAlternativeNameQualifier.object_type
      • SubjectAlternativeNameQualifier.format_reference_human_readable()
      • SubjectAlternativeNameQualifier.model_post_init()
    • X509Certificate
      • X509Certificate.expires_in
      • X509Certificate.issuer
      • X509Certificate.object_type
      • X509Certificate.pk_algorithm
      • X509Certificate.pk_number
      • X509Certificate.pk_size
      • X509Certificate.serial_number
      • X509Certificate.signed_by
      • X509Certificate.subject
      • X509Certificate.valid_from
      • X509Certificate.valid_until
      • X509Certificate.format_reference_human_readable()
      • X509Certificate.model_post_init()
      • X509Certificate.expired
  • octopoes.models.ooi.monitoring
    • Application
      • Application.name
      • Application.object_type
      • Application.model_post_init()
    • Incident
      • Incident.application
      • Incident.event_id
      • Incident.event_title
      • Incident.event_type
      • Incident.meta_data
      • Incident.object_type
      • Incident.severity
      • Incident.model_post_init()
  • octopoes.models.ooi.question
    • Question
      • Question.json_schema
      • Question.object_type
      • Question.ooi
      • Question.schema_id
      • Question.json_schema_valid
      • Question.model_post_init()
      • Question.config_pk
  • octopoes.models.ooi.network
    • AutonomousSystem
      • AutonomousSystem.name
      • AutonomousSystem.number
      • AutonomousSystem.object_type
      • AutonomousSystem.model_post_init()
    • IPAddress
      • IPAddress.address
      • IPAddress.network
      • IPAddress.format_reference_human_readable()
      • IPAddress.model_post_init()
    • IPAddressV4
      • IPAddressV4.address
      • IPAddressV4.netblock
      • IPAddressV4.object_type
      • IPAddressV4.model_post_init()
    • IPAddressV6
      • IPAddressV6.address
      • IPAddressV6.netblock
      • IPAddressV6.object_type
      • IPAddressV6.model_post_init()
    • IPPort
      • IPPort.address
      • IPPort.object_type
      • IPPort.port
      • IPPort.protocol
      • IPPort.state
      • IPPort.format_reference_human_readable()
      • IPPort.model_post_init()
    • IPV4NetBlock
      • IPV4NetBlock.mask
      • IPV4NetBlock.object_type
      • IPV4NetBlock.parent
      • IPV4NetBlock.start_ip
      • IPV4NetBlock.model_post_init()
    • IPV6NetBlock
      • IPV6NetBlock.mask
      • IPV6NetBlock.object_type
      • IPV6NetBlock.parent
      • IPV6NetBlock.start_ip
      • IPV6NetBlock.model_post_init()
    • NetBlock
      • NetBlock.announced_by
      • NetBlock.description
      • NetBlock.name
      • NetBlock.network
      • NetBlock.parent
      • NetBlock.format_reference_human_readable()
      • NetBlock.model_post_init()
    • Network
      • Network.name
      • Network.object_type
      • Network.format_reference_human_readable()
      • Network.model_post_init()
    • PortState
    • Protocol
  • octopoes.models.ooi.reports
    • AssetReport
      • AssetReport.input_ooi
      • AssetReport.object_type
      • AssetReport.report_type
      • AssetReport.format_reference_human_readable()
      • AssetReport.model_post_init()
    • BaseReport
      • BaseReport.data_raw_id
      • BaseReport.date_generated
      • BaseReport.name
      • BaseReport.observed_at
      • BaseReport.organization_code
      • BaseReport.organization_name
      • BaseReport.organization_tags
      • BaseReport.reference_date
      • BaseReport.report_recipe
      • BaseReport.template
    • HydratedReport
      • HydratedReport.input_oois
      • HydratedReport.object_type
      • HydratedReport.report_type
      • HydratedReport.format_reference_human_readable()
      • HydratedReport.model_post_init()
      • HydratedReport.to_report()
    • Report
      • Report.input_oois
      • Report.object_type
      • Report.report_type
      • Report.format_reference_human_readable()
      • Report.model_post_init()
    • ReportData
      • ReportData.data
      • ReportData.object_type
      • ReportData.organization_code
      • ReportData.organization_name
      • ReportData.organization_tags
      • ReportData.format_reference_human_readable()
      • ReportData.model_post_init()
    • ReportRecipe
      • ReportRecipe.asset_report_types
      • ReportRecipe.cron_expression
      • ReportRecipe.input_recipe
      • ReportRecipe.object_type
      • ReportRecipe.recipe_id
      • ReportRecipe.report_name_format
      • ReportRecipe.report_type
      • ReportRecipe.model_post_init()
  • octopoes.models.ooi.dns.records
    • CAATAGS
    • DNSAAAARecord
      • DNSAAAARecord.address
      • DNSAAAARecord.dns_record_type
      • DNSAAAARecord.object_type
      • DNSAAAARecord.model_post_init()
    • DNSARecord
      • DNSARecord.address
      • DNSARecord.dns_record_type
      • DNSARecord.object_type
      • DNSARecord.model_post_init()
    • DNSCAARecord
      • DNSCAARecord.dns_record_type
      • DNSCAARecord.flags
      • DNSCAARecord.object_type
      • DNSCAARecord.tag
      • DNSCAARecord.value
      • DNSCAARecord.model_post_init()
    • DNSCNAMERecord
      • DNSCNAMERecord.dns_record_type
      • DNSCNAMERecord.object_type
      • DNSCNAMERecord.target_hostname
      • DNSCNAMERecord.model_post_init()
    • DNSMXRecord
      • DNSMXRecord.dns_record_type
      • DNSMXRecord.mail_hostname
      • DNSMXRecord.object_type
      • DNSMXRecord.preference
      • DNSMXRecord.model_post_init()
    • DNSNSRecord
      • DNSNSRecord.dns_record_type
      • DNSNSRecord.name_server_hostname
      • DNSNSRecord.object_type
      • DNSNSRecord.model_post_init()
    • DNSPTRRecord
      • DNSPTRRecord.address
      • DNSPTRRecord.dns_record_type
      • DNSPTRRecord.object_type
      • DNSPTRRecord.format_reference_human_readable()
      • DNSPTRRecord.model_post_init()
    • DNSRecord
      • DNSRecord.dns_record_type
      • DNSRecord.hostname
      • DNSRecord.ttl
      • DNSRecord.value
      • DNSRecord.format_reference_human_readable()
      • DNSRecord.model_post_init()
    • DNSSOARecord
      • DNSSOARecord.dns_record_type
      • DNSSOARecord.expire
      • DNSSOARecord.minimum
      • DNSSOARecord.object_type
      • DNSSOARecord.refresh
      • DNSSOARecord.retry
      • DNSSOARecord.serial
      • DNSSOARecord.soa_hostname
      • DNSSOARecord.format_reference_human_readable()
      • DNSSOARecord.model_post_init()
    • DNSTXTRecord
      • DNSTXTRecord.dns_record_type
      • DNSTXTRecord.object_type
      • DNSTXTRecord.model_post_init()
      • DNSTXTRecord.natural_key
    • NXDOMAIN
      • NXDOMAIN.hostname
      • NXDOMAIN.object_type
      • NXDOMAIN.format_reference_human_readable()
      • NXDOMAIN.model_post_init()
  • octopoes.models.ooi.dns.zone
    • DNSZone
      • DNSZone.hostname
      • DNSZone.object_type
      • DNSZone.parent
      • DNSZone.format_reference_human_readable()
      • DNSZone.model_post_init()
    • Hostname
      • Hostname.dns_zone
      • Hostname.name
      • Hostname.network
      • Hostname.object_type
      • Hostname.registered_domain
      • Hostname.format_reference_human_readable()
      • Hostname.hostname_valid
      • Hostname.model_post_init()
    • ResolvedHostname
      • ResolvedHostname.address
      • ResolvedHostname.hostname
      • ResolvedHostname.object_type
      • ResolvedHostname.format_reference_human_readable()
      • ResolvedHostname.model_post_init()
  • octopoes.models.ooi.software
    • Software
      • Software.cpe
      • Software.name
      • Software.object_type
      • Software.version
      • Software.format_reference_human_readable()
      • Software.model_post_init()
    • SoftwareInstance
      • SoftwareInstance.object_type
      • SoftwareInstance.ooi
      • SoftwareInstance.software
      • SoftwareInstance.format_reference_human_readable()
      • SoftwareInstance.model_post_init()
      • SoftwareInstance.natural_key
• Octopoes Research
  • Introduction
  • Part I - Requirements, context and inherent complexities
    • Context of Octopoes in KAT
      • Event loop
      • KAT dataflow
    • Objectives
      • KAT
      • Octopoes
      • Desired/future
    • Complexities of Octopoes
      • Open-world assumption
      • Contradicting information
      • Logic
      • Reasoning
      • Consistency
      • Logical programming
      • Computing inferences and materialization
      • What knowledge can we actually infer from our objects?
      • Knowing things
      • Not knowing things
      • Knowing that you don’t know things
      • Other assorted notes and thoughts
  • Part II - Working towards a solution
    • Stages of data processing
      • Data model / domain of discourse
      • Upfront DDL?
      • Entity ID
    • ClaimSpace
    • FactSpace
    • Inference
    • Handling updates / incoming data
      • Batch processing
      • Incremental processing / streaming
• Reports
  • Creating reports
    • Location of the report code
    • Steps to create a new report
    • Collecting data
  • Writing report unit tests
    • Purpose of unit testing
    • Steps for writing unit tests
    • Executing unit tests
• Development tutorial
  • Glossary
    • Creating a Boefje
      • Creating a Boefje Variant
      • Creating your own OCI image
      • Creating an OCI image using our Base Image
    • Testing the boefje
    • Creating a new model
    • Creating a normalizer
      • normalizer.json
      • normalize.py
    • Creating a bit
      • __init__.py
      • bit.py
      • check_greeting.py
    • Creating a report
      • __init__.py
      • report.py
      • report.html
      • Conclusion
• QA Test plan
  • Read the PR
  • Manual testing
  • Check the Docker logs
  • Document QA notes
  • On occasion