Using eBPF to Enhance Monitoring and Observability
By Nate Matherson, Co-founder & CEO of ContainIQ
The ability to inject code into Linux kernels opens up a whole new world of possibilities. There is so much that you can improve easily—security, network, observability, and more. BPF (Berkeley Packet Filter) enables you to write programs that can leverage the power of the Linux kernel from within. BPF has been traditionally used to filter raw network packets before sending them to userspace to increase the system’s overall security.
eBPF is an extended version of BPF with an array of security implementations to prevent BPF programs from breaking the kernel. In this guide, you will learn how you can use eBPF to implement enhanced observability into your Kubernetes-based infrastructure.
What is eBPF?
eBPF stands for extended BPF. As goes its name, It is an extended version built on top of Linux’s standard BPF observability system. eBPF programs are attached to the Linux code path via triggers (or hooks). One of the prime objectives here is not to make any changes to the kernel source code to reduce the chances of dealing any damage to it.
You can view it as a virtual machine or a sandbox within the Linux kernel that enables you to access specific kernel resources while restraining itself from making changes to the kernel itself. eBPF also takes special care of safety and robustness whenever crossing userspace boundaries.
There are a number of ways in which eBPF enforces security constructs. Here are a few examples:
- The in-kernel verifier will not load an eBPF program if it detects an invalid pointer dereference or the maximum stack size limit is reached.
- Loops are not allowed, except those with predefined static upper bounds
- Only a tiny subset of eBPF helper functions is allowed to be called within the resulting bytecode.
With all these restrictions in place, one might find those programs limiting. However, the assurance of not messing up with the kernel usually trumps these limitations.
How Can eBPF Be Used to Help In Monitoring?
eBPF programs are run directly in Linux kernels. This makes it easy for them to trace almost any aspect of the OS’s subsystems, including CPU scheduler, networking, system calls, etc. You can view and track nearly everything that happens in the operating system via eBPF programs. This makes it a viable contender to set up global and deep-rooted monitoring in your Linux-based deployments.
Moreover, eBPF programs do not depend on any external modules or dependencies. Therefore, the performance overhead that such programs add to the standard workload of Linux programs is negligible. Running a monitoring system on it adds virtually no additional load to your deployed workload.
Another aspect to ponder over is the rise of container-optimized operating systems. Such operating systems, including Container-Optimized OS by Google Cloud Platform, provide an optimized running environment for K8s apps; however, they are designed mostly immutable to reduce code footprint and enhance scaling. They also do not allow kernel modules to be utilized. eBPF still works in this case since it is native to Linux and provides a way to extract data without depending on kernel modules at all.
4 Benefits of Using eBPF To Enhance Monitoring
Now that you understand how It works and plays an integral role in Kubernetes monitoring, here are a few ways in which it enhances your monitoring efforts.
Pro: Low Intrusiveness
It focuses on minimalism when interacting with the kernel and its resources. It aims to be as less intrusive as possible. Here are some ways in which it does so:
- It does not alter any kernel source code, which reduces the coupling between eBPF programs and the kernel.
- When leveraging it as a debugger, it does not require halting a running program to observe its state.
- Only a tiny subset of kernel helper functions are allowed, further reducing the dependency/coupling between the two and helping towards a sandboxed environment for running your programs.
Pro: Security
The above measures also contribute towards enhancing the security of the kernel and your programs.
Since eBPF programs are not allowed to modify the kernel at all, your access management rules for code-level changes can be preserved easily. Otherwise, this would have been done using a kernel module, which would bring a plethora of security concerns.
Also, eBPF programs are run through a verification phase after being converted into bytecode. This step checks for resource leakages present in the program, such as infinite loops. This verification prevents a program from accidentally over-using the available resources on the system.
Pro: Convenience
It provides you with greater convenience with your monitoring efforts. You get more granular details and kernel context than any other Linux monitoring alternative. You also get the freedom to export the monitoring data and ingest it into third-party visualization tools.
Aside from that, if you take a look at the concept of BPF programming from a high-level perspective, it makes much more sense and is easier to add code snippets that are hooked into kernel functions compared to maintaining independent kernel modules.
Pro: Programmability
Compared to standard instrumentation techniques, It gives you room to run operations on the logged data before dumping it. You can choose to store data between eBPF events and selectively store data as needed. This feature improves the overall utility of your monitoring system without adding additional dependencies.
4 Specific Use Cases For Using eBPF?
Here are four specific use-cases in which it would be the best alternative for gaining deep visibility.
Kubernetes Observability
Kubernetes is one of the best places to demonstrate the prowess of eBPF for observability. As is known, Kubernetes scales a workload up and down by increasing or decreasing the number of pods. The pods are random in nature and have no guaranteed lifecycle. Hence, setting up instrumentation agents in each pod or container can introduce performance issues in the workload as well as not turn out to be effective since the pods will be created and destroyed randomly.
Using it, you can set up your monitoring camp on the OS level and monitor everything that happens with your Kubernetes setup.
Dynamic Network Performance Monitoring
We have already mentioned multiple times how eBPF can monitor and trace internal aspects of the Linux subsystems, such as CPU usage, network performance, etc., quite easily. You can make use of this trait of eBPF to create a network performance monitoring setup. However, such a setup would be static since the rules used to monitor the network are predefined constants. You can still change them manually, but that is a lengthy process.
To make this system truly dynamic, you can introduce Cilium, an open-source project that helps you with rule management in eBPF. Cilium can help you analyze network traffic usage packet by packet and adjust the eBPF rules automatically to adapt to the incoming workload requirements. You can leverage it to create a truly dynamic network performance monitoring setup.
Kernel Tracing
Reiterating how eBPF works—it allows you to run custom code inside the Linux kernels. Since all activities occurring in a system are done in the kernel, it becomes easy to track and trace everything from one place. You can also set up the eBPF programs to be executed based on system events. This enables you to track everything that occurs around such events.
It is also one of the safest ways to implement kernel tracing since all eBPF programs are passed through a verification phase where they are checked for infinite loops and other error possibilities. Therefore, you can safely rely on it for your kernel tracing requirements.
Pod-Level Network Monitoring
One of the most popular uses of eBPF lies in monitoring the network around the pods in a Kubernetes-based system. Since a Kubernetes-based setup can have a variety of running applications, with each of them having its own unique base images, it is pretty challenging to instrument them with conventional methods. Each variation of the base operating system, cloud server, or coding standard might require a different monitoring agent.
Implementing network monitoring via eBPF can help you get around these issues. Moreover, you can carry out socket filtering at the Linux cgroup level. eBPF programs will help you get granular information that would otherwise be accessible only in the /sys directory in Linux. Since you have kernel access via eBPF, you can receive more detailed information along with context from the kernel.
Cloudanix Workload Security
Secure your Kubernetes based container workloads across AKS, EKS and GKE.
Secure your Kubernetes now!Final Thoughts
eBPF is an excellent addition to the traditional BPF mechanism. It provides you with added security benefits while enabling the deep observability benefits of traditional BPF programs.
In this guide, we talked about how you can use eBPF for enhancing observability in your Kubernetes setup. We discussed some of the benefits of using eBPF over traditional BPF and mentioned a handful of use cases too where you can leverage it for the best results.
GCP Kubernetes Monitoring
Your number of Kubernetes clusters can grow pretty fast - and so could be your pain points.
Know more about:
- Why Do We Need Continuous Audits For Public Cloud.
- Top 16 AWS S3 Misconfigurations To Avoid In 2022.
- How Can Your Application Accomplish ISO27001 in AWS Cloud?
Insights from Cloudanix
