In-Depth Review: How Accurate Are Today’s SBOM Tools
Introduction
As the spotlight on software supply chain security intensifies, the call for adopting Software Bill of Materials (SBOM) becomes more resounding. To aid developers in this critical task, several SBOM generation tools have emerged, with Trivy and Syft leading the pack. These tools, boasting thousands of stars on GitHub and widespread integration in CI/CD pipelines, promise to simplify the generation of SBOMs. However, the burning question remains: How well do they perform? To answer this question, we conducted a thorough evaluation, documented in a comprehensive white paper. This blog serves as a condensed overview, highlighting notable issues discovered during our evaluation. For a more in-depth exploration, readers are encouraged to delve into the white paper.
Tested Tools
In this blog, we delve into a comprehensive case study involving four prominent SBOM generation tools: Trivy 0.43.0 1, Syft 0.84.1 2, Microsoft SBOM Tool 1.1.6 3, and GitHub Dependency Graph 4. Through this exploration, we aim to shed light on their SBOM analysis results and dissect the reasons behind the observed differences.
Finding 1: Supported Formats - The Foundation of SBOM Generation
At the core of SBOM generation lies the extraction of metadata which contains crucial dependency information. A fundamental aspect to evaluate is the supported metadata formats. Without compatibility, these tools risk overlooking essential details. The table below showcases the supported metadata formats for each tool:
| Trivy | Syft | sbom-tool | GitHub DG | ||
|---|---|---|---|---|---|
| Golang | go.mod | ✓ | ✓ | ✓ | ✓ |
| Go executable | ✓ | ✓ | |||
| Java | pom.xml | ✓ | ✓ | ✓ | |
| gradle.lockfile | ✓ | ✓ | ✓ | ✓ | |
| MANIFEST.MF | ✓ | ✓ | |||
| pom.properties | ✓ | ✓ | |||
| JavaScript | package.json | ✓ | ✓ | ✓ | ✓ |
| package-lock.json | ✓ | ✓ | ✓ | ✓ | |
| yarn.lock | ✓ | ✓ | ✓ | ✓ | |
| pnpm-lock.yaml | ✓ | ✓ | ✓ | ||
| PHP | composer.json | ✓ | |||
| composer.lock | ✓ | ✓ | ✓ | ||
| Python | requirements.txt | ✓ | ✓ | ✓ | ✓ |
| poetry.lock | ✓ | ✓ | ✓ | ✓ | |
| pipfile.lock | ✓ | ✓ | ✓ | ✓ | |
| setup.py | ✓ | ✓ | ✓ | ||
| Ruby | Gemfile | ✓ | |||
| Gemfile.lock | ✓ | ✓ | ✓ | ✓ | |
| .gemspec | ✓ | ✓ | ✓ | ✓ | |
| Rust | Cargo.toml | ✓ | |||
| Cargo.lock | ✓ | ✓ | ✓ | ✓ | |
| Rust executable | ✓ | ✓ |
Finding 2: Promises vs. Reality - Discrepancies in Metadata Support
Despite claims of comprehensive metadata support, both Trivy and Syft fall short of expectations. Notably, their documentation asserts that package.json parsing is supported. However, in practice, both tools fail to return dependencies declared in package.json. This discrepancy raises concerns as the absence of vital dependency information significantly diminishes the value of the generated SBOM, leaving developers with a compromised view of their software supply chain.
Sample package.json:
{
"author": {
"name": "Pascal Belloncle",
"url": "http://jsdox.org"
},
"maintainers": [
{
"name": "Marc Trudel",
"email": "mtrudel@wizcorp.jp",
"web": "http://www.wizcorp.jp"
}
],
"license": "MIT",
"name": "jsdox",
"description": "Simple JSDoc 3 to Markdown generator",
"version": "0.4.10",
"repository": {
"type": "git",
"url": "git://github.com/sutoiku/jsdox.git"
},
"main": "./jsdox.js",
"files": [
"bin/jsdox",
"jsdox.js",
"LICENSE",
"README.md",
"templates",
"lib"
],
"scripts": {
"lint": "node_modules/.bin/jscs jsdox.js lib test && node_modules/.bin/jshint --config .jshintrc jsdox.js lib/* bin/* test/* fixtures/*",
"test": "npm run lint && node_modules/.bin/mocha;"
},
"preferGlobal": true,
"bin": {
"jsdox": "./bin/jsdox"
},
"dependencies": {
"jsdoc3-parser": "^1.0.3",
"mustache": "^0.8.2",
"optimist": "~0.3.4",
"q": "^1.0.1"
},
"devDependencies": {
"expect.js": "^0.3.1",
"jscs": "^1.6.1",
"jshint": "^2.5.3",
"mocha": "^1.21.4",
"sinon": "^1.10.3"
},
"optionalDependencies": {},
"engines": {
"node": ">=0.8.4"
},
"homepage": "http://jsdox.org",
"bugs": "https://github.com/sutoiku/jsdox/issues"
}
Output:
{
"$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json",
"bomFormat": "CycloneDX",
"specVersion": "1.5",
"serialNumber": "urn:uuid:626cbf3d-3abe-49ff-b6f1-2f294b387f39",
"version": 1,
"metadata": {
"timestamp": "2023-11-14T16:35:35+00:00"
},
"components": [],
"dependencies": [
{
"ref": "e553c696-c615-4c22-adec-18ae720bd844",
"dependsOn": []
}
]
}
Finding 3: Incomplete Results
In our testing, three scenarios led to incomplete SBOM results: naive parsing, handling of development/testing dependencies, and transitive dependencies.
-
Native parser:
Trivy and Syft encounter challenges when parsing certain metadata formats, such as
requirements.txtin Python. Trivy and Syft rely on splitting each line by the double equal sign ("=="). However, this approach fails to detect cases where developers use package names only (defaulting to the latest version) or specify version ranges, a common practice in Python development. Trivy and Syft's inability to handle such nuances results in overlooked dependencies.Sample requirements.txt
django>=1.9.0 django-crispy-forms>=1.6.0 django-import-export>=0.5.1 django-reversion>=2.0.0 django-formtools==1.0 future==0.15.2 httplib2==0.9.2 six==1.10.0Output:
{ "$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json", "bomFormat": "CycloneDX", "specVersion": "1.5", "serialNumber": "urn:uuid:91492c06-4b1e-4915-900f-2be39dfc44eb", "version": 1, "metadata": { "timestamp": "2023-11-14T17:18:04+00:00" }, "components": [ { "bom-ref": "6381e123-81f4-4af6-8f19-49d3c8d1b694", "type": "application", "name": "requirements.txt", "properties": [ { "name": "aquasecurity:trivy:Class", "value": "lang-pkgs" }, { "name": "aquasecurity:trivy:Type", "value": "pip" } ] }, { "bom-ref": "pkg:pypi/django-formtools@1.0", "type": "library", "name": "django-formtools", "version": "1.0", "purl": "pkg:pypi/django-formtools@1.0", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "pip" } ] }, { "bom-ref": "pkg:pypi/future@0.15.2", "type": "library", "name": "future", "version": "0.15.2", "purl": "pkg:pypi/future@0.15.2", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "pip" } ] }, { "bom-ref": "pkg:pypi/httplib2@0.9.2", "type": "library", "name": "httplib2", "version": "0.9.2", "purl": "pkg:pypi/httplib2@0.9.2", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "pip" } ] }, { "bom-ref": "pkg:pypi/six@1.10.0", "type": "library", "name": "six", "version": "1.10.0", "purl": "pkg:pypi/six@1.10.0", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "pip" } ] } ], "dependencies": [ { "ref": "6381e123-81f4-4af6-8f19-49d3c8d1b694", "dependsOn": [ "pkg:pypi/django-formtools@1.0", "pkg:pypi/future@0.15.2", "pkg:pypi/httplib2@0.9.2", "pkg:pypi/six@1.10.0" ] }, { "ref": "ec5ede3a-0efd-4ff3-9404-0056c0260230", "dependsOn": [ "6381e123-81f4-4af6-8f19-49d3c8d1b694" ] }, { "ref": "pkg:pypi/django-formtools@1.0", "dependsOn": [] }, { "ref": "pkg:pypi/future@0.15.2", "dependsOn": [] }, { "ref": "pkg:pypi/httplib2@0.9.2", "dependsOn": [] }, { "ref": "pkg:pypi/six@1.10.0", "dependsOn": [] } } -
Development and testing dependencies: Some metadata formats lack native support for specifying different sets of dependencies for distinct purposes. For instance, JavaScript's
package.jsonusesdevDependenciesto distinguish development dependencies. In contrast, Python developers often use different file names likerequirements_dev.txt. Trivy sticks to the originalrequirements.txtonly, while Syft employs heuristics to accommodate diverse naming conventions. However, there is no “correct” approach for this because including development dependencies in SBOM may introduce false alarms. -
Transitive dependencies: Trivy, Syft, and GitHub Dependency Graph do not resolve transitive dependencies, which are dependencies of dependencies. As many metadata formats lack explicit specifications for transitive dependencies, this oversight can result in false negatives, compromising the thoroughness of the SBOM.
Finding 4: Inconsistent Naming
In cases where package names comprise multiple parts, such as Java package names with group id and artifact id, different SBOM tools adopt varying strategies for composing package names.
Given
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>32.1.3-jre</version>
</dependency>
Syft reports guava as the package name, Microsoft SBOM tool uses com.google.guava**.**guava (dot concatenation) whereas Trivy and GitHub Dependency Graph use com.google.guava**:**guava (colon concatenation). This inconsistency introduces challenges in interpreting and correlating data across different tools.
Finding 5: Version
Package versioning is crucial for vulnerability detection, but the SBOM tools exhibit divergent behaviors. Golang, for example, prefixes versions with a leading letter "v" (e.g., v1.9.0). While Syft and Microsoft SBOM Tool adhere to this convention, Trivy and GitHub Dependency Graph omit this letter during SBOM assembly. Additionally, GitHub Dependency Graph fails to resolve version constraints specified as ranges (e.g., >= 1.2.0 or ^5.9.0), reporting them verbatim in the SBOM file.
Example:
{
"SPDXID": "SPDXRef-DOCUMENT",
"spdxVersion": "SPDX-2.3",
"creationInfo": {
"created": "2023-11-14T17:21:28Z",
"creators": ["Tool: GitHub.com-Dependency-Graph"],
"comment": "Exact versions could not be resolved for some packages. For more information: https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#dependencies-included."
},
"name": "com.github.requirejs/requirejs",
"dataLicense": "CC0-1.0",
"documentDescribes": ["SPDXRef-com.github.requirejs-requirejs"],
"documentNamespace": "https://github.com/requirejs/requirejs/dependency_graph/sbom-0aa8a5b3f183f407",
"packages": [
{
"SPDXID": "SPDXRef-com.github.requirejs-requirejs",
"name": "com.github.requirejs/requirejs",
"versionInfo": "",
"downloadLocation": "git+https://github.com/requirejs/requirejs",
"filesAnalyzed": false,
"supplier": "NOASSERTION",
"externalRefs": [
{
"referenceCategory": "PACKAGE-MANAGER",
"referenceType": "purl",
"referenceLocator": "pkg:github/requirejs/requirejs"
}
]
},
{
"SPDXID": "SPDXRef-npm-jscs",
"name": "npm:jscs",
"versionInfo": "^ 1.12.0",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"supplier": "NOASSERTION"
},
{
"SPDXID": "SPDXRef-npm-jshint",
"name": "npm:jshint",
"versionInfo": "^ 2.7.0",
"downloadLocation": "NOASSERTION",
"filesAnalyzed": false,
"supplier": "NOASSERTION"
}
],
"relationships": [
{
"relationshipType": "DEPENDS_ON",
"spdxElementId": "SPDXRef-com.github.requirejs-requirejs",
"relatedSpdxElement": "SPDXRef-npm-jscs"
},
{
"relationshipType": "DEPENDS_ON",
"spdxElementId": "SPDXRef-com.github.requirejs-requirejs",
"relatedSpdxElement": "SPDXRef-npm-jshint"
}
]
}
Finding 6: Parser Confusion Attack
A parser confusion attack exploits inconsistencies between different parsers processing the same input, allowing malicious actors to craft deceptive input that’s benign for one parser but harmful for another. Take Python requirements.txt for example:
requests [security]>=2.8.1,==2.8.*
Output:
{
"$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json",
"bomFormat": "CycloneDX",
"specVersion": "1.5",
"serialNumber": "urn:uuid:90478287-d48b-401e-9b6b-306946a3d874",
"version": 1,
"metadata": {
"timestamp": "2023-11-14T17:31:52+00:00"
},
"components": [
{
"bom-ref": "9bf57409-e5fa-4834-9b84-db1c39a6ab5c",
"type": "application",
"name": "requirements.txt",
"properties": [
{
"name": "aquasecurity:trivy:Class",
"value": "lang-pkgs"
},
{
"name": "aquasecurity:trivy:Type",
"value": "pip"
}
]
},
{
"bom-ref": "pkg:pypi/requests%3E%3D2.8.1%2C@2.8.%2A",
"type": "library",
"name": "requests\u003e=2.8.1,",
"version": "2.8.*",
"purl": "pkg:pypi/requests%3E%3D2.8.1%2C@2.8.%2A",
"properties": [
{
"name": "aquasecurity:trivy:PkgType",
"value": "pip"
}
]
}
],
"dependencies": [
{
"ref": "20401769-1f9d-479d-a7b5-0366561f2ccc",
"dependsOn": ["9bf57409-e5fa-4834-9b84-db1c39a6ab5c"]
},
{
"ref": "9bf57409-e5fa-4834-9b84-db1c39a6ab5c",
"dependsOn": ["pkg:pypi/requests%3E%3D2.8.1%2C@2.8.%2A"]
},
{
"ref": "pkg:pypi/requests%3E%3D2.8.1%2C@2.8.%2A",
"dependsOn": []
}
]
}
In this example, Trivy finds a package called requests\u003e=2.8.1, of version 2.8.* which does not exist. On the other hand, pip install "requests [security]>=2.8.1,==2.8.*" installs requests without any issue. This effectively hides packages from automated scanning. This underscores the importance of addressing parser inconsistencies to maintain the integrity of SBOM results and mitigate potential security risks.
Summary
In essence, this blog encapsulates key issues encountered in popular SBOM generation tools. For a nuanced understanding and detailed insights, the complete findings are documented in our white paper. Additionally, we offer a free product online, providing developers with accessible resources to enhance their software supply chain security practices.