Author: Eric Guo, DevOps Director, Aurora
Introduction
My company Aurora Health Science & Technology Co., Ltd (“Aurora”) has an exciting mission: we enable life science clinical research and development, leveraging AI for faster and more accurate results. As a data company, we need to take care of protecting our customer’s data assets in our system. Real-time analysis of container vulnerabilities can help us take actions at once when critical vulnerabilities are found. Leveraging open-source and commercial products, my company, Aurora set up a system which can provide actionable insights for our DevOps team to keep our system secure at all times.
There are three main components to my setup:
Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain. Dependency-Track takes a unique and highly beneficial approach by leveraging the capabilities of Software Bill of Materials (SBOM). This approach provides capabilities that traditional Software Composition Analysis (SCA) solutions cannot achieve.
Trivy is a simple and comprehensive vulnerability/misconfiguration/secret scanner for containers and other artifacts. Trivy detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and language-specific packages (Bundler, Composer, npm, yarn, etc.). In addition, Trivy scans Infrastructure as Code (IaC) files such as Terraform and Kubernetes, to detect potential configuration issues that expose your deployments to the risk of attack. Trivy also scans hardcoded secrets like passwords, API keys and tokens. Trivy is easy to use. Just install the binary and you’re ready to scan. All you need to do for scanning is to specify a target such as an image name of the container.
Timeplus is a streaming-first real-time analytics platform. It enables fast analytics with ultra-low latency, while ensuring high EPS (events-per-second) both in ingestion and query simultaneously. In early testing, Timeplus can achieve 4ms end-to-end latency, and 10 million + EPS benchmark, in a single commodity machine. It includes powerful real-time streaming analytics that enable functionality such as windowing/non-windowing, late event, downsampling and streaming predictive analytics, all from one SQL query. With Timeplus, you can easily connect to diverse data sources and immediately explore streaming patterns via query and visualization, and create real-time multi-channel notifications and send aggregated data to downstream systems.
Use Case: Building real-time analytics for supply chain management
As global supply chains have become increasingly sophisticated, industry software has become more complex. This has led to an increase software vulnerabilities to cyber attacks, which are becoming much more common among organizations worldwide.
Aurora’s clinical platform is based on micro-service architecture, it adopts large amounts of open-source components such as docker images and libraries. Applications are packaged within docker containers, so it is important for Aurora’s security team to understand which components are adopted and identify the risk of such components as early as possible, to avoid potential risks caused by security issues in those images and libraries.
Based on this requirement, we set up a system which leverages Trivy and Timeplus to do real-time risk/vulnerabilities analysis and trigger alerts to take necessary action as needed. For example, when CVE-2022–2992 was found on October 18 2022, the real-time system alerted our DevOps team immediately, and we contacted our supply provider to take prompt action.
Technical Details
1. Data source
SBOM (Software Bill of Materials) as a data source of software metadata which includes system libs and application third party dependencies. Currently there are two popular standard formats, such as SPDX and CycloneDX.
SPDX is an open standard for communicating Software Bill of Material information, including components, licenses, copyrights, and security references. SPDX reduces redundant work by providing a common format for companies and communities to share important data, thereby streamlining and improving compliance.
OWASP CycloneDX is a lightweight Software Bill of Materials (SBOM) standard designed for use in application security contexts and supply chain component analysis. You can refer https://cyclonedx.org/docs/1.4/json/ for more details.
In this solution, we use Trivy to generate CycloneDX formatted SBOM file and then sink to Timeplus to do real-time analysis:
# Generate SBOM file
trivy image — security-checks vuln -f cyclonedx -o trivy-cyclonedx.cdx atlassian/jira-software:8.20.14Part of SBOM file:
{
"metadata": {
"timestamp": "2022–11–03T07:22:16+00:00",
"tools": [
{
"vendor": "aquasecurity",
"name": "trivy",
"version": "0.34.0"
}
],
"component": {
"bom-ref": "pkg:oci/jira-software@sha256:cc9f9142231195cf509dae9600429d20ed80823250fc57ae3b51b90e2c8a2f7a?repository_url=index.docker.io%2Fatlassian%2Fjira-software\u0026arch=amd64",
"type": "container",
"name": "atlassian/jira-software:8.20.14",
"purl": "pkg:oci/jira-software@sha256:cc9f9142231195cf509dae9600429d20ed80823250fc57ae3b51b90e2c8a2f7a?repository_url=index.docker.io%2Fatlassian%2Fjira-software\u0026arch=amd64",
"properties": [
{
"name": "aquasecurity:trivy:SchemaVersion",
"value": "2"
},
{
"name": "aquasecurity:trivy:ImageID",
"value": "sha256:d67cf82a6b104ee30c3637b9f5c42d6cb41642784d9fb0aabb117ec0f405b5b9"
},
{
"name": "aquasecurity:trivy:RepoDigest",
"value": "atlassian/jira-software@sha256:cc9f9142231195cf509dae9600429d20ed80823250fc57ae3b51b90e2c8a2f7a"
},
{
"name": "aquasecurity:trivy:DiffID",
"value": "sha256:f4a670ac65b68f8757aea863ac0de19e627c0ea57165abad8094eae512ca7dad"
},
{
"name": "aquasecurity:trivy:DiffID",
"value": "sha256:974e434b1b00fd93e57245daf74dc30b02ea3ad08e433d3ffe3b67e6dbc86a97"
},
{
"name": "aquasecurity:trivy:RepoTag",
"value": "atlassian/jira-software:8.20.14"
}
]
}
}
}2. Sink SBOM to Timeplus
In the code below, we can sink the SBOM file to Timeplus. Timeplus has the ability to abstract json format, so we can store metadata, components and vulnerabilities as text format.
def output_timeplus(sbom, provider):
api_key = os.environ.get("TIMEPLUS_API_KEY")
env = (
Env().schema("https").host("timeplus.yaochn.com").port("443").api_key(api_key)
)
Env.setCurrent(env)
# Create a stream, ref https://docs.timeplus.com/docs/datatypes
try:
stream = (
Stream()
.name("aurora_security")
.column(StreamColumn().name("provider").type(Type.String))
.column(StreamColumn().name("image").type(Type.String))
.column(StreamColumn().name("bomFormat").type(Type.String)) .column(StreamColumn().name("specVersion").type(Type.String))
.column(StreamColumn().name("serialNumber").type(Type.String)) .column(StreamColumn().name("version").type(Type.Integer))
.column(StreamColumn().name("metadata").type(Type.String))
.column(StreamColumn().name("components").type(Type.String))
.column(StreamColumn().name("dependencies").type(Type.String))
.column(StreamColumn().name("vulnerabilities").type(Type.String))
)
if (stream.get() is None):
stream.create()
except BaseException as err:
sys.exit(f"Failed to list or create data streams from https://timeplus.yaochn.com:443. Please make sure you are connecting to the right server. {err=}, {type(err)=}")
with open(sbom) as fh:
sbom_data = json.load(fh)
results = []
result = [provider, sbom_data["metadata"]["component"]["name"], sbom_data["bomFormat"], sbom_data["specVersion"], sbom_data["serialNumber"], sbom_data["version"],
json.dumps(sbom_data["metadata"]), json.dumps(sbom_data["components"]), json.dumps(sbom_data["dependencies"]), json.dumps(sbom_data["vulnerabilities"])]
results.append(result)
stream.insert(results)3. SBOM stream search
select * from aurora_security settings seek_to='earliest'
From the above screen, we can get all scan events, every scan includes multiple vulnerabilities, for example:
4. Powerful JSON processing ability
One SBOM includes multiple vulnerabilities. For analysis, we need to abstract vulnerabilities and explode those vulnerabilities. Let’s do it as below:
select p, i, array_join(a) as vul, vul:id as id from(
select provider as p,
image as i,
vulnerabilities as raw,
json_extract_array(raw) as a
from aurora_security
settings seek_to='earliest'
) where p='atlassian'
As above, we can see that one container includes 115 vulnerabilities.
5. Dedup
As a company, we do continuous image scans to find the latest vulnerabilities, except we need to understand vulnerabilities within one image. We also need to know vulnerabilities company wide, so we need to dedup vulnerability with a continuous image scan:
select distinct(id) as vul_id from (
select p, i, array_join(a) as vul, vul:id as id from(
select provider as p,
image as i,
vulnerabilities as raw,
json_extract_array(raw) as a
from aurora_security
settings seek_to='earliest'
) where p='atlassian')
Results
Lastly, using the Timeplus dashboard, we set up a monitoring dashboard which provides a whole picture of container vulnerabilities. With this dashboard, we can easily understand all of the critical vulnerabilities of suppliers:
Summary
For DevOps engineers, identifying software vulnerabilities as early as possible is mission critical. In today’s software environment, where dependencies are more and more complex, we’ve found that using powerful tools like Trivy and Timeplus makes real-time monitoring systems easy to set up without compromising on functionality or latency. Leveraging these tools, Aurora has set up a powerful, low latency, real-time vulnerabilities analysis system.
Note: All vulnerability data are generated by Trivy and can be obtained easily with public solutions.