How and why we built the Kubernetes Vulnerability Scanner

It started with a conversation between Daniel Bechenea, our Lead Engineer & Product Manager, and myself, after a launch. We discussed the possibility of integrating a new scanner into our platform, one that would find vulnerabilities in Kubernetes clusters. We felt this was a hot topic for our current days, one that our product didn’t address yet.

At first, we had multiple debates about where we could integrate this (as another part of Network Scanner, as another part of Cloud Scanner or as a brand, new tool). Also, we discussed what kind of attack scenarios we want to take on, and how we should start working on this (starting from scratch vs integrating an open-source tool and improving on it).

After we decided what we wanted to build, or at least had a vague idea, we started researching the topic to see what is the exact methodology behind keeping a Kubernetes cluster secure, and how we could help cybersecurity specialists in doing so.

Does infosec really need another Kubernetes vuln scanner?

When building a new tool, we always look at the bigger picture to understand what fellow cybersecurity practitioners need. 

We knew the increasing reliance on cloud technologies amplifies the need for container orchestration solutions. Kubernetes has emerged as the industry standard and it’s transformed how applications are deployed and managed in the cloud-native environment. We see this in our own product infrastructure too. 

Plus, KubeCon and the growth of the Cloud Native Computing Foundation, which now oversees over 180 projects with 240,000 contributors, also make it glaringly obvious that Kubernetes is becoming a major part of our global tech infrastructure.

What gives orchestrators like Kubernetes even more power is that they manage credentials and allocate resources for all services. As a result, it only makes sense that they’re a target for attackers, as they hold tremendous power in the cluster they control. A high impact attack means that malicious actors can leak confidential data, terminate processes, reallocate resources, and run remote commands in the entire infrastructure. Even without full privileges, exploiting misconfigurations can lead to serious breaches, including enhanced coordination of Denial of Service attacks. For example, the exposed /pods/ endpoint, might leak secret tokens and passwords, to remote unauthenticated attackers.

Given Kubernetes' popularity and the pivotal role of orchestrators, they are prime targets for attackers. The cryptojacking attack on Tesla’s Kubernetes infrastructure is a prime example of that. It's imperative to secure each attack surface according to the unique demands of the infrastructure, especially for environments that are constantly evolving, such as cloud-native platforms or hybrid environments, where traditional security tools may not suffice.

Our goal with building a dedicated Kubernetes Scanner was to give red teams a reliable tool for rigorously testing these infrastructures to eliminate as many vulnerabilities as possible and prevent potential exploits. And we also wanted to make it as easy to use as possible, so busy DevSecOps teams can also make sure insecure configurations, and vulnerable components don’t make it into production.

 What our Kubernetes Scanner does that others can’t

We began developing the Kubernetes Scanner with a focus on black and gray box remote scanning scenarios, as these are the most common among bug bounty hunters, pentesters, and red-teamers. Our daily experience confirms this, and our services team can attest that these types of engagements represent the majority of their work.

Because we know offensive security folks need to identify vulnerabilities, misconfigurations, and exposed control panels with a vulnerability scanner focused on Kubernetes infrastructures, we started with the open-source Kube-Hunter tool, which now is unmaintained. We chose it because it’s the most popular and thorough Kubernetes scanning tool that focuses on the use case we chose. Its successor, Trivy, is also very popular, but focuses on a wider range of use cases, without adding much to the black and gray box remote scenario. 

Here’s how we dealt with Kube-Hunter’s limitation, building our own tool to overcome them and deliver the reliable capabilities our customers - fellow infosec practitioners - need.

Overcoming port scanning limits

Kube-Hunter only scans a preset list of ports, but we wanted to allow our customers to scan any ports they want, as certain targets might not expose services only on their default ports. We also wanted to offer some preset list of ports to customers who aren’t sure which ports they should look into when scanning a Kubernetes cluster.

So we built our Kubernetes Scanner to offer the following port configurations:

• Top 10 ports that expose Kubernetes specific services

• Top 100 ports that expose Kubernetes specific services

• Full port range

• Any list or interval of ports the user wants to scan

To determine the first two presets, we analyzed the distribution of ports from the Kubernetes clusters listed on Shodan. Based on this insight into real-world Kubernetes clusters we were able to build a comprehensive list of ports that help cybersecurity professionals scan targets fast, while not compromising the accuracy of the scan.

Overcoming assumptions when detecting exposed services

If the port that a Kubernetes-specific service (e.g. Etcd, Kubelet API, etc.) uses as default is open, Kube-Hunter assumes the port must expose the service.. 

However, this is not always the case, and often non-default ports are used to expose services. 

We decided to solve this by creating custom probing mechanisms to detect the services. This means our scanner decides if a service is exposed or not based on the responses it gives to certain requests instead of simply going by the port number.

Improving on limited reporting capabilities

Whenever we build a tool, we always develop it with reporting in mind, since reports are what reflect our customers’ work. We knew our Kubernetes Scanner - just like all our vulnerability tools - needed to generate ready-to-deliver reports in just a couple of seconds, after a scan finishes.

The first step was to adjust the risk levels for all findings to the severities we found fit. To do so, we used the CVSS version 3. 

Then, we added risk descriptions to all the findings. These need to clearly describe to which risks a vulnerability exposes the affected target. This field provides excellent value for the report reader, particularly executives. It can also make the vulnerability’s impact more straightforward for the technical people reading it. 

To match our reporting standard, we also added an evidence field to all findings, showing how to reproduce them and how we find them. This helps customers to validate a finding at a glance and ensure it’s not a false positive. We believe any report needs to include the steps to reproduce the issue, so we provide them directly instead of letting customers do extra work to find out how they can replicate the vulnerability. 

In addition to all of the above, we added a Confirmed/Unconfirmed tag to all findings. These tags show our level of certainty, and the finding we generate is a true positive. 

For example, suppose we claim a target is vulnerable to a CVE based only on its version. In that case, we might be wrong for reasons such as the version not being reported correctly or the version being vulnerable but patched. In this case, we want to attach an unconfirmed tag to the finding. On the other hand, if we claim we managed to gain Remote Code Execution on the target and we confirmed the command ran by verifying its output, we can be sure the finding is factual.

Expanding detection to reliably cover more Kubernetes vulnerabilities

We also added new detections to the set Kube-Hunter already packed. For example, we added detection for the cAdvisor service, which can hold valuable information in the context of a Kubernetes cluster, such as container metadata, logs and resource usage (which can be leveraged for DOS attacks).

Giving access to real-time results

Lastly, but not least, we changed the mechanism through which a finding gets sent to the user, making it show findings in real-time. 

This means that our customers can see our Kubernetes Scanner’s findings in real-time, instead of waiting for them until the scan ends.

How everything came together

To better test our Kubernetes Scanner and Kube-Hunter, we decided to compare their results against two targets. 

The targets exposed services for which both tools have detections: Etcd, Kubernetes API, Kubelet API). Both targets exposed the same services with the same vulnerabilities, but on different ports.

Target 1 exposed all services on their default ports, while target 2 exposed Etcd on its default port, and the other two services on non-default ports.

Severity distribution across vulnerabilities

Scanning both targets with the Pentest-Tools.com Kubernetes Vulnerability Scanner, we can see from the screenshot  that the number of findings and their distribution across severities is identical.

Kube-Hunter output

However, ten out of eleven findings disappeared from one scan to another when scanning the same two targets using Kube-Hunter, as we can see from the screenshot above. Furthermore, Kube-Hunter fails to find the service even in the setup that uses one of the most common ports for the Kubernetes API Server. 

Conversely, our Kubernetes Vulnerability Scanner not only discovers the service but also finds a low-severity finding on it.

In addition to all the above, besides using the Kubernetes Vulnerability Scanner as a standalone tool, our customers can leverage all features of Pentest-Tools.com as a product. 

For example, you can use our Network Scanner to hunt for vulnerabilities in the services that your cluster exposes. For instance, when the Network Scanner finds a Remote Code Execution CVE, use it to get access to a pod in the cluster. If you find a CVE that leads to sensitive information disclosure, use it to get critical information about the cluster you are targeting, such as service account tokens or environment variables. You can also combine scheduled scans and our scan diff feature to implement continuous monitoring for your assets.

Feel free to give Kubernetes Vulnerability Scanner a try. Its light mode version is available to all our users, or you can get on a paid plan to unleash its full power.

What it’s like to build a specialized vulnerability scanner and next steps

Building this tool I learned a lot, first and foremost, about Kubernetes, of course. I also learned more about what it's like to think about our tools from a macro perspective. My previous work at Pentest-Tools.com was focused on writing exploits and detections and improving our tools, but I never had to think about a tool’s architecture from scratch before. This was interesting, and my colleagues proved to be of great help when making decisions, as we spent countless hours discussing what would be the best approach.

Multiple challenges popped up when building this tool. From creating a playground environment to test our detections, to even simpler decisions I hadn’t thought about before, such as how the findings should be propagated to the front-end.

We believe our Kubernetes Vulnerability Scanner is a state of the art improvement for its category of tools, but we don’t plan to stop here. We have a range of improvements in mind, from new detections and exploits to better integrations with other tools that will make this scanner an even more important asset for our customers.

I appreciate you taking time to go behind the scenes with me! Let me know what else you’re interested in finding out about how and why we build things at Pentest-Tools.com.