Getting features in the main branch
Most features should follow the path laid out on our project board. This document describes the requirements for features to move between columns of the board. Listing these requirements should reduce the bystander effect for doing code or QA reviews and make it easier for other developers to pick up these tasks. This would enable us to move features quickly from in progress to merged and avoid bottlenecks at either the review or QA stage. The required procedure to merge a feature into main are as follows.
1. Approved Features / Need Refinement → Refined Tasks
We are not reinventing the wheel: there is no high-quality library that already has this feature.
This issue is “bite-sized” and (only) leaves non-critical implementation details to the developer.
2. In Progress → Review
The authors of the ticket have created a pull request.
Checklists for authorsin our pull_request_template has been filled in by the authors.
3. Review → QA review
Checklist for code reviewersin our pull_request_template has been filled in by a code reviewer.
4. QA Review → Ready for Merge
Checklist for QAin our pull_request_template has been filled in by a QA reviewer.
5. Ready for merge → Done
The procedures above should guarantee that members of the kat-managers group can merge these features directly. We should actively aim to resolve any discussions about the implementation at stages 1 and 2. It is the responsibility of the authors to bring possible issues to the attention of anyone that might have an opinion about the issue.
Once a release branch has been created with a new set of functionality, it is important that we do a QA review again. This time, the QA has an extended checklist to also guarantee that there is no regression in the more advanced functionality of OpenKAT. Also, we need to assure that there is no regression between the different supported deployments options of OpenKAT.
Environments for the extended QA
Clone the source repository and run
make reset[Linux and Darwin, perhaps different docker versions and installs]
Install the debian packages [On different distro’s: ubuntu 20.04 + 22.04, debian 11 + 12]
Install the container images
Ideally we would follow the following QA procedure on each of these environments:
Checklist for QA
I confirmed that there are no unintended functional regressions in this branch:
I have managed to pass the onboarding flow
Objects and Findings are created properly
Tasks are created and completed properly
Extended checklist for QA
Checking the UI/UX
Turning Boefjes on and off in the KATalogus
Create, turn off, and delete Boefjes-settings
Reports (Findings), per object, per report
Pagination of several tables
Manually starting Boefjes and normalizers
Manually adding and deleting objects and Findings
Automatic scheduling and starting of Boefjes and normalizers
Exporting the object list as JSON and CSV
Inspection of task details
Inspection of all pages interfaces, including de tree- and graph view of objects
UI/UX in general
Checking User/Organization management functionality
I can create and delete an organization
I can create and delete users
I can assign and revoke rights to these users
I can reset 2FA
Verify that there is no significant performance regression
Tips and tricks for pull request QA testing
Think outside the box
Feel free to deviate from the checklist: testing things that are not obviously related to the PR is a good way to find bugs.
Thoroughness is key: embrace the “hacker mindset” and try to break (new) functionality by providing unexpected input, and attempt to perform unauthorized actions.
Try to break the UI: try resizing the window, using zoom functionality, and test multiple browsers.
Always remember that you are taking on the role of a user that is probably not as familiar with the application as you are: everything you encounter should feel intuitive and easy to use. Lack of intuitiveness deserves a QA comment.
Be pragmatic but versatile
Features updating the data model should usually be backward compatible, so we should not run
make resetupon every review. Switch tactics with respect to updating your local environment regularly.
Small documentation changes do not require rebuilding and restarting all services to performa a QA review.
Properly gauge the impact of a feature: API changes in the KATalogus, for example, can affect Rocky, Mula and Octopoes, but never Bytes (in the current setup).
Changes that hit the core of every service (package updates) require performing the extended QA checklist.