Kubernetes defines several interfaces (CNI, CSI, and CRI) to make different parts of the system extensible. This blog briefly discusses CNI and, more specifically, how Sarena uses it. It is not the intention of this blog to describe all the ins and outs of CNI or cover every possible edge case.

The Container Network Interface (CNI) is a simple contract between a container runtime and a network plugin. Its purpose is to configure network connectivity for containers and to clean up that connectivity when a container is removed.

I my previous post, Basic eBPF forwarding, I asked the question:

“What happens if the destination IP isn’t in the lxc_map?”

The answer in that post was basically: the packet gets dropped.

To summarize: The lxc_map is a node-wide map that maintains a mapping from a destination-ip to an EndpointInfo. The EndpointInfo contains an interface index. So, when the desination-ip is known, the eBPF program simply redirects it to that interface.

This post is about (very) basic pod-to-pod networking using eBPF. The goal is to create two endpoints, wire them together into an eBPF dataplane, and then send a UDP packet from one to the other to prove that the whole path works using the bare minimum.

The scope is deliberately narrow: same-L2-segment, same-host pod-to-pod forwarding. There is no overlay, cross-host routing, IPv6, policy, conntrack, or upstream connectivity. Those are future work. For now, the goal is to build the smallest possible dataplane that can get one packet from one pod to another.

This post is about a design problem that shows up in many eBPF-based networking project sooner or later, and largely independent of what the dataplane itself actually forwards: how do you attach, track, and tear down kernel-resident programs from a userspace control plane, in a way that survives your own process dying at the worst possible moment?

The project this is drawn from is an early-stage eBPF/XDP/TC dataplane written in Rust on top of aya. The packet-forwarding logic itself — routing, ARP, policy — is still mostly stubbed out. What is built out, and what this post is about, is the layer underneath all of that: the loader that manages the lifecycle of eBPF programs and their pinned kernel state, independent of whatever those programs eventually do.

This framework is heavily inspired by Cilium’s eBPF testing approach. The overall testing architecture comes from Cilium; this post focuses on implementing those ideas in Rust using Aya. If you’re interested in the original design, I recommend taking a look at Cilium’s testing framework. The implementation described in this post is available in my GitHub repository.

• Cilium: https://github.com/cilium/cilium
• This implementation: https://github.com/erwin-kok/sarena

The problem

When exploring/implementing an eBPF/XDP/TCX dataplane, a question arose: “How do we actually test this?” For example, when we have an eBPF program called process_ipv4(), how can we test its functionality? Granted, these programs are written in Rust (when using Aya), so it looks straightforward to test them like any other Rust function. However, there are a few caveats:

Anyone writing eBPF programs for long enough will eventually have a conversation with the verifier. While developing several eBPF programs, I ran into a number of verifier rejections that were not immediately obvious from the source code. In this article I discuss two examples that required looking beyond the Rust source and into the generated BPF bytecode.

The verifier’s contract

The kernel verifier performs static analysis on every BPF program before it is loaded. Its job is to prove that the program is safe to execute under the rules of the eBPF execution model.