The Shellshock vulnerability demonstrated is a serious vulnerability, but it's just one of thousands of known, exploitable vulnerabilities that range from negligible right up to critical in severity. Most organizations have a policy of not deploying software with known vulnerabilities over a certain severity threshold.
A vulnerability scanner can show you which vulnerabilities are present in your container images. This data can inform policies to prevent deploying vulnerable containers that an attacker could exploit.
There are several solutions for vulnerability scanning available, and in this workshop we will use the open source scanner Trivy.
Running Trivy locally¶
Let's start by installing Trivy so we can run it locally.
On MacOS using Homebrew:
On MacOS / Linux:
Trivy does not have a native binary for Windows, but you can run Trivy as a container.
For more details see the installation instructions.
Scan images with Trivy¶
To scan an image simply run
trivy image <image>. For example, to find the vulnerabilities in the image used in the Shellshock-compromised pod, run this:
The Shellshock vulnerability is classed as HIGH severity, and it has the identifier CVE-2014-6271.
If you are seeing a lot of results you might want to filter out the lower severity results. For example:
For comparison, try this version of the image that does not have the Shellshock vulnerability:
Now try Trivy on your favourite images!
Vulnerability scanning in CI/CD¶
One way to "shift left" security is to include vulnerability scanning as an automatic step in CI/CD. Your build steps might be something like this:
- Compile code
- Run tests
- Build Dockerfile into container image
- Scan image for vulnerabilities
- Push image to container registry
If any step fails, the pipeline is stopped - so if a vulnerability is found, the scanning stage fails, and the image doesn't get pushed to the registry.
Trivy has some settings that make it a great fit in CI/CD systems:
--lightdownloads a smaller vulnerability database that omits some details such as the text description of each vulnerability
--severity=CRITICAL,HIGH,MEDIUMonly checks for vulnerabilities of MEDIUM or higher severity
--exit-code=1returns 1 if any vulnerabilties are found (above the threshold defined by
--severity). This non-zero exit code is interpreted as a failure by the CI/CD system.
Scanning in admission control¶
To prevent deploying a container image with known vulnerabilities, you can use an admission controller that either
- scans the image, or
- retrieves the result from a previous scan
and denies admission if the image is vulnerable.
Scanning in live workloads¶
Scanning in the CI/CD pipeline can prevent you from storing a vulnerable image in the registry, and admission control can prevent you from deploying an image with known vulnerabilities. But you may also want the ability to check the container images used by your live workloads.
Security is an ongoing process
New vulnerabilities are found all the time. An image that you scan today might not have any vulnerablities, but a researcher may find a new issue that means the same image will fail the scan tomorrow.
Starboard is a tool for running security tools, including Trivy, within your Kubernetes cluster. This is an easy way to create and view scans of the container images used by your running workloads. Want to give it a try?
If you don't already have the
krewplugin manager installed, follow the instructions here
Install Starboard as a kubectl plugin:
- Check that it's working
On Mac you may get warnings about not being able to verify the developer - see this guide to resolve them.
- Initialize Starboard
- Check that this has created some CRDs
Starboard uses Trivy to populate some
vulnerabilities CRs that relate to running workloads. Let's find the vulnerabilities in the deployment we created for the Shellshock demo.
This will take a few seconds. The
--delete-scan-job=false parameter allows us to inspect the job that Starboard creates to run Trivy. But first let's look at the results:
This outputs all the vulnerability information (including the details about Shellshock, with ID CVE-2014-6271).
These details have been stored in a
vulns resource that you can view with a regular
kubectl get command:
The output should look something like this:
The labels indicate which resource this vulnerability report applies to.
Upcoming feature: additional info
On master, but not yet released as a binary, we have
additional info using
-o wide option.
As mentioned above, Starboard creates a Kubernetes job that runs Trivy over the container images defined for the workload. This job lives in a namespace called starboard. In this example, the job finds that the
shellshockable deployment uses the container image
lizrice/shellshockable:0.1.0, and runs Trivy over that image. If you specified
--delete-scan-job=false you can inspect the completed job.
Within this output you can see that the job ran the
trivy command, and the last argument to that command was the image name
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
View in Octant¶
Starboard makes it easy to run Trivy over your running workloads, and coming soon, there will be an operator that will watch for new workloads and automatically run scans over them.
Today, there is also an Octant plugin so you can view the vulnerability results through the Octant UI.
You can check out the instructions for this here if you want to try it out.