Kubernetes Attacks: What is Your Cluster Trying to Tell You?

Pawan Shankar: Hello, everyone. Welcome to Kubernetes Attacks: What Your Cluster is Trying to Tell You.

My name is Pawan Shankar, and I'm responsible for product marketing here at Sysdig. Excited to be here with you, Brad.

Brad Geesaman: Likewise. And I'm Brad Geesaman, co-founder of Darkbit. We're here to talk about ways to improve security detection in your Kubernetes clusters without breaking the bank.

Organizations that have adopted a DevOps culture tend to follow a repeatable pattern for shipping software. First, they get it working, then they deploy through some form of automation. Ops teams then implement just enough operational monitoring, and then there's pressure to rinse and repeat. After gaining confidence with this pattern, that's typically when some security measures get introduced. For example, code security and vulnerability scans just before new deployments in the pipeline. This can often add unwanted friction.

When additional security checkpoints are added, this often increases friction and slows overall velocity. The delicate, nuanced balance of risk reduction versus feature velocity tends to consume a lot of resources from all of the teams involved. And this leaves very little time for focusing on follow-on security activities. As we've seen firsthand, many organizations have underdeveloped container security detection and response capabilities.

For this talk, malicious activity detection means that when a security incident happens inside the cluster, that activity is captured. All the activities are centrally exported to another location, and the activities are then filtered to form high-confidence indicators, and the teams are notified with supporting context.

Why is this capability so uncommon? Well, there's a couple contributing factors. Number one, defaults have inertia. Kubernetes defaults are unlikely to be sufficient in terms of supporting security detection needs. When the default configurations are baked into production, the risk and cost of changing them later is much higher.

DevOps and infrastructure teams are typically busy shipping and keeping things running operationally, and that's where they're incentivized. Security teams often lack sufficient expertise about containerized environments and common attack scenarios in them. There's actually an education gap. Finally, there's a perception of low risk due to infrequency of security issues. That said, most clusters lack sufficient defense-in-depth measures that would prevent even a rare security incident from causing serious damage.

Security of the environment is a shared responsibility, and if DevOps and infrastructure teams are focused on shipping, and security teams don't have the understanding of what to look for, and there's no pressing security event forcing this focus, we end up with a gap in that middle ground that attackers can take advantage of.

So how might attackers take advantage of that gap in responsibility? While it's not an exhaustive list, here are four common attack scenarios that Kubernetes operators should be aware of. First, a publicly available API server with misconfigured permissions or even a vulnerability. There are actually bots that scan for this every day. Exploitable flaws in libraries or applications, such as Apache Struts or a custom code vulnerability in your actual application code. Third, leaked developer or administrative credentials. This is AWS keys, SSH keys, API keys pushed into public code repositories, or even just plain old phishing attacks against those developer laptops. And lastly, via vulnerable or malicious code dependency in the supply chain getting baked into the image. This could be a Node npm module, a Python pip module, a Go module, a RubyGem, et cetera.

We're going to focus on supply chain compromise. There are two common ways to bypass the trust in containerized software supply chains. First, you create a dependency and get other packages to use it, and those then get baked into an image. And second, you just create a malicious image and entice others to run it.

So we asked, if a malicious container image is run inside of a Kubernetes cluster, what first few steps might a knowledgeable attacker take? And critically, would we even know about it?

We created a default Kubernetes cluster for our first demo environment. As it turns out, not much is available to us for detection without additional effort. The API audit logs, which are activities of access and changes to Kubernetes resources, who made them, and when, actually requires additional configuration to enable. The Kubernetes component logs, logs from the processes that make up Kubernetes itself, the API server, the kubelet, kube-proxy, etcd, et cetera, those are actually logged to local files on the nodes. And logs from containerized workloads are logged locally to files on the nodes, but they're also not shipped anywhere off the cluster by default. And Kubernetes doesn't have any in-container malicious behavior logs. That's just not part of what its responsibilities are.

So essentially, we lack key kinds of visibility unless we SSH into the nodes and specifically look for items or take additional measures to collect those logs.

You may have heard about automated attacks that mine bitcoins as the desired goal of the attacker, but let's be a little bit more creative. Let's run a container image with a malicious dependency via a deployment manifest. When the developer deploys that container in the cluster, that's step one. The malicious image is pulled from the public registry. That's step two. We'll run a snippet of malicious code. That's step three. And just enough code to fetch and then run a shell script from my attacker-controlled web server. That's step four. And finally, in step five, the actual payload. It'll download some tools inside the container, attempt to enumerate permissions against the Kubernetes API server, and then if it can, create a separate malicious workload that runs on all worker nodes. If that succeeds, that workload will upload all of the secrets from all of the Kubernetes worker nodes to that same attacker web server.

Now, there's a couple of assumptions. The container image is publicly available, and outbound access from the cluster to the internet is allowed, which is the default. And the permissions for the default service account allow the creation of other pods. So let's see that in action.

So the attacker has slipped the malicious code snippet into a code dependency. Let's take a look at that. And as you can see at the top here where it's commented out, it's obfuscated in the bottom. But at the top, we're just pulling in a script from a web server, turning it into a shell script, making it executable, and executing it. And that gets baked into a Docker container.

So let's look at the Dockerfile that makes that Docker container. And right here in this line where we add the contents of that app folder into the container, that's where it gets baked in. And we've created this malicious container called Brad Geesaman Ruby app with the tag v123.

So when this container runs, it downloads and runs a shell script. So let's look at that shell script here. First, we're going to install curl if we can, using a package manager. Using curl, we're going to attempt to download kubectl, the binary, inside the container. We're going to use kubectl to list secrets or try to connect to the API server to list secrets.

Next, we're going to enumerate some permissions. So in the list auth, we're going to ask the API server what permissions do we have, and we're going to send them back to our attacking web server. And lastly, if we have the ability to create a pod, we're going to attempt to run a DaemonSet, again pulling that DaemonSet from the attacking web server.

So if that container has Kubernetes API access, it runs that secret-stealing DaemonSet. So let's look at that manifest. And this is what would be pulled down in that script. What it's doing here is a loop, and it's mounting the host file system, and it's looking for all the Kubernetes secrets that are attached to the worker nodes. Again, this is run against all of the nodes in the cluster.

On our attacking web server, we're going to be looking at the web access logs. So I'll start that tail now. And if we're a developer and we're interacting with this default cluster, this is what this cluster will look like. It's a very basic kubeadm cluster with one node, and it has some of the basic pods in here. There's no real workloads just yet.

So now we're the developer, and we're going to deploy the Ruby app that we talked about that has this malicious dependency baked in. As you can see, it's very straightforward. Nothing malicious on the surface. So when they deploy that manifest with a kubectl apply, seems like nothing happened, right?

Let's go back over to the attacking web server, and we can see that it asked for the shell script. It posted the results of: did it have access? And then it went and grabbed the DaemonSet manifest and applied it. And all of these posts you see here are from the pods running in that DaemonSet, sending all the secrets back to the attacking web server. So in less than 10 seconds, all of the secrets have been exfiltrated. There's no API audit logs. There's no malicious activity detection logs, no network logs by default in cloud environments. Really, we have no visibility.

We can summarize what just happened in a more general way using the MITRE ATT&CK framework terminology. In other words, a shared language that your defenders will likely understand. This demo mimics the following realistic attack techniques. Number one, we got a shell inside a container. We launched a package manager in the container. That was to install curl. We contacted the Kubernetes API server from the container multiple times to enumerate permissions to start the privileged DaemonSet, which brings us into the next step. We started the privileged container. That privileged container then searched for keys and passwords and launched a suspicious network tool to send those back to our attacking web server.

So if we want to see all of this activity, should we just get all the logs possible? Charity Majors of Honeycomb.io answers that pretty succinctly. The answer is no, not everything. It's a great way to spend a lot of money and drown in busy work, and it doesn't actually improve your detection. A methodical approach is needed that balances the detection capability with time and cost.

So with that methodical approach, it's important to focus on the right areas. David Bianco published his Pyramid of Pain in 2013 in a great blog post, and it describes how to build detections that are more painful for the attackers to bypass. So instead of chasing millions of MD5 hashes of binaries and IP addresses, things that attackers can easily bypass or sidestep, we focus on the tactics and techniques at the top of the pyramid that describe the attacker behaviors. In other words, we're focusing on detections with high signal and low noise because high-quality detections become higher-quality events, which become higher-quality alerts.

Let's look at the available Kubernetes security logs through this new lens. A few have high signal, but two stand out as also having low noise and a reasonable volume. Specifically, the API server audit logs and in-container malicious behavior logs.

If we look back at our four attack scenarios, we can see several places where attackers may share common behaviors, and we can look at how well those behaviors are covered by just these two log sources. So in purple, we can see that the API server audit logs can tell us who or what interacts with the API server, and those types of actions relating to permissions enumeration and getting all the Kubernetes secrets, they really stand out. And in teal, we can see activities happening inside containers themselves can clue us in to when data is accessed, exfiltrated, or leaked, and when workloads with extra privileges to the nodes are run, and certain types of attacker tool usage.

So if we resort this list, we can see which of these log types can provide a high degree of security value without requiring a large investment of time and energy. Let's enable these two log streams in a new demonstration cluster and see what this visibility affords us now.

We created a default Google Kubernetes Engine, or GKE, cluster for this demo. We've enabled a few specific logs, and we ship them to a central location, specifically the API server audit logs, the Kubernetes component logs, and containerized workload logs. We've also installed Falco, an open-source container security detection suite donated to the CNCF by Sysdig, and that's going to alert when malicious activity happens inside a containerized workload. All of these logs are all shipped off the cluster to the cloud provider's logging system.

We've then used some light filtering to pick these logs out of the stream and drop them into a single location for consolidated viewing, so we don't have to SSH into any virtual machines.

So the new cluster has Falco installed and Fluent Bit to ship the logs centrally. Here we can see with the kubectl get pods, there's a few more things going on here. But basically, the Falco namespace at the very top shows that the Falco DaemonSet is installed on this one-node cluster, and the Fluent Bit DaemonSet is installed shipping all of those logs.

So if we refer back to that same manifest, this rubyapp.yaml, again, it's the same thing, no changes made. And if that developer deploys that manifest into this new GKE cluster, again, there's nothing obvious that just happened to the developer. However, we can see all the secrets are being sent and posted from this cluster. There they go. So they're all there. That just happened.

However, tailing the raw logs from the Falco DaemonSet shows us a lot has just happened. Falco has picked up on a number of things. But this isn't really easy to look at, right? Let's go look at it where it looks a lot easier to parse.

Over here, these are the filters that I talked about. We're just picking out a couple key things: all of the Falco alerts, the listing of secrets that fail, creation of privileged DaemonSets, and some of the permissions enumeration. And this ends up into this log stream here, where we can see the package management adding curl. We could see the moving of kubectl into the binary directory. We can see the getting of secrets.

And right here where I've highlighted, you can see that the API server audit logs have showed there's a permission denied on the list secrets. A couple of instances where the kubectl binary is used to connect to the API server and run inside the container. We could see the enumeration of permissions from this default service account. Again, another access to the API server. We could see where the ask was: can we create a pod specifically? We could see where those come from the kubectl running. And lastly, we can see the sensitive mount container where the privileged DaemonSet ran and sent all the secrets back to our web server.

So if we look at our log viewer of these filtered events, we can now trace each step and know really within a few seconds that this cluster was compromised, which pod to investigate, and which malicious actions were taken.

Pawan Shankar: And what Brad just showed is how effective it is to enable a few of these key audit log type environments, such as the Kubernetes audit logs, and also using the in-container malicious activity that you're monitoring for. And both of these serve as great tools in your runtime security toolbox. So now let's dive into a little bit more detail as to how you could implement strong runtime security in your environment.

The way to get that deep visibility from inside your containers, one approach is to leverage the system calls. Now, the Sysdig agent is deployed on your host, and either via an eBPF probe or installing an open-source kernel module, we look inside the kernel and have visibility into all the system calls that are traversing the kernel. That could be your host network metrics, other full-stack metrics, custom metrics like Prometheus, and all security events that might be happening in your environment. All of this data is collected via the Sysdig agent, and we store that in our backend.

Now, what do we do with that data? So we collect all that granular data and enrich it with the metadata from your cloud and your Kubernetes environments. So whether you're running in Kubernetes or other multi-cloud environments, we can slice and dice that data that we collect and allow you to see all that deep data from an application and a service lens. So now your development team can not just identify a vulnerability in Kubernetes, but map it back to a specific namespace or a service that might be affected.

So beyond just vulnerabilities, you can also detect and monitor for changes such as your CPU or other memory changes. And a lot of times, I/O changes can be a great indicator of compromise. So being able to map that back to specific services and applications allows you to respond faster to a breach that might be occurring in your environment.

Now, Brad talked a lot about audit logs and how it's a great way to get that high signal-to-noise. Integrating with audit logs via Falco and Sysdig is very simple. So you may have users or workloads writing API calls that get registered in the Kubernetes audit log events, and then all of these audit logs are automatically ingested by the Sysdig agent. And we have out-of-the-box policies that allow you to write detection rules based on events that are happening and registered in the audit log.

Some examples could be: did someone store credentials in a config map versus secrets? Did someone exec into a pod, modify a file? Where was that connection initiated from? As well as, were there some privilege escalation or permission changes that might be happening in your Kubernetes environments? Really understanding who did what in Kubernetes can come from the API server, and that audit log gives you a great audit trail, essentially, of exactly what happened. All of these events are then sent back to the Sysdig platform, and you can also analyze this with full detail and put a lens of your applications, your Kubernetes environments, to really pinpoint exactly what's happening and where it's happening.

Finally, what happens when something does go wrong? The question all security teams are asking is, do we have a forensics plan in place? And especially when it comes to containers, containers are ephemeral by nature, which means a malicious attacker can compromise a container, and the container could be long gone or spun down by Kubernetes. So really having that audit trail that gives you the visibility across the user level, all the way down to the events that happened. So what commands, the connections that were made, all of that data and audit trail is something that your team needs.

And then you can also use tools like Sysdig Inspect, an open-source tool, to really go deep into the forensics data that you might be collecting. So very similar to Wireshark, where you had a PCAP file, you can generate an SCAP file, which just takes a dump of all the system calls and allows you to recreate all that system activity and really go deep into your incident response workflows and pinpoint exactly what happened even after that container is long gone. So this allows you to have a robust incident response and forensics plan in place to respond to events when they do occur in your environment.

So time for a story. Let's talk through an example of one customer that had leveraged the audit log integration and found it extremely useful when it came to enforcing the strong runtime security in their environment. And this was a company that was looking to track sensitive modifications that happened inside their cluster. They were already using open source tools, loved Falco, and really wanted to be able to leverage and extend that in their environment. So some of the detection rules that came with Sysdig Secure that were out of the box really helped them save time in creating some of these rules from scratch.

And with the Sysdig Secure detection engine that's built on Falco, it allowed them to create flexible policies on top of all that audit log information. And then when potentially a sensitive modification happened, they could understand: was this a malicious event? Was this a routine operation? And ultimately, just having more visibility into exactly what's going on in their cluster and making their detection response workflows much more efficient and much faster when it came to events that were occurring in their environment.

So we talked a lot about audit logs, runtime security inside your containers. Now what about your cloud? Because a lot of times these containers are living in your AWS, Google Cloud, and other environments. So when it comes to AWS specifically, we're extending this detection capability to secure not just your containers, your Kubernetes clusters, but also your AWS cloud environments.

So when we think about CloudTrail, it's very similar to how we were talking about audit logs, where a lot of times AWS users and services are performing these API calls across a large number of AWS services. I think there are close to 175 services, and this list continues to grow. So CloudTrail becomes a great security tap where all these services are sending their logs out to.

You can collect the logs from CloudTrail and ingest that into Sysdig. Again, similar to how you were writing Falco-based detection rules on top of the audit logs from Kubernetes, similarly here, you're able to write those detection rules on top of the CloudTrail logs in AWS. And this allows you to detect whether an S3 bucket has encryption turned off. Did someone launch a load balancer that's public-facing or change some IAM role permissions? All these events, again, are logged there. You can write detection rules to be alerted on it immediately without wading through a ton of logs trying to find that needle in a haystack. And in the Sysdig platform, you can apply these policies as well as see all those results in a centralized place, or if your preference is to forward them to AWS Security Hub or other tools, you can do that as well.

So just to summarize, the Sysdig Secure DevOps platform is really built on an open source foundation, leveraging Falco, our open source contribution to the CNCF, Prometheus for monitoring, Encore Engine, and the Sysdig open source. And the two key products here are Sysdig Secure and Sysdig Monitor, and they're tied together in a single platform because we believe that ultimately, if cloud teams want to ship applications faster, they should embed security and monitoring into their DevOps workflow.

Some use cases that we support as customers are adopting Kubernetes and containers in their production environments. Starting off with the essential workflows, we package them up in a tier called Essentials, and that provides five key workflows: image scanning, runtime security, container and Kubernetes monitoring, cloud services monitoring, and compliance. And as teams start to get more mature, they're looking for some of the advanced workflows that are offered in the enterprise tier. So native prevention and enforcement via pod security policies, helping you generate and validate that before you deploy them in production, machine learning-based image profiling, the incident response and the forensics workflow, as well as more troubleshooting and extended compliance coverage that are offered there as well.

Finally, we're really integrated into your cloud-native stack. Starting from the build phase, we integrate directly into your CI/CD pipelines or your registry, so we can scan your images where they are. And we extend this further by offering an inline scanning approach, which allows you to scan locally on the same node or in your registry without sending images outside of your environment. And this is a more secure approach because you're not sharing your sensitive data or your registry credentials to a third-party vendor. All the scanning happens locally, and the only thing that's sent back to Sysdig is the metadata about the scanned results.

Runtime security, again, built around Falco and also more monitoring capabilities by extending Prometheus. And then from a response standpoint, really plugging into the tools that your developers love, such as PagerDuty, Slack, ServiceNow, as well as forwarding events to tools like Splunk, Syslog, IBM QRadar, really having a strong response framework that is aligned with the tools that you're already using. And then the platform, again, can be deployed self-hosted in your own environment or offered as a SaaS service. For many DevOps teams, this is very critical because their teams are small and there's not a lot of people that can maintain the product. So having a SaaS service relieves them of that and allows them to focus on solving the core security use cases that the product provides.

So, just to sum it back up here, what Brad talked about was a lot of reasons for why it's difficult to detect malicious activity. And a couple of the ways that you can solve for that is really collecting that valuable security logs via the audit logs, via the in-behavior activity that's happening using Falco, and really allows your DevOps and security teams to understand exactly what's happening in your environment.

And we also talked about how the Sysdig platform can extend the capabilities of the audit logs using Falco, with more workflows and processes around that, that allows you to have a better handle between the security and the DevOps team, and ultimately apply that shared responsibility model that Brad talked about earlier inside your container environments by shining that light on any malicious activity that's happening at runtime.

And if this was useful and you're interested in more resources, we have a list here. Definitely check those out, and you can learn more about all of these topics in more detail. And if there are any more questions, jump into the private Slack channel for the Ask Us Anything session. It's at expo.sysdig. We'd love to engage in more conversation and answer any questions that you guys might have.

Thank you so much. Appreciate the time.