Host routing

I my previous post, Basic eBPF forwarding, I asked the question:

“What happens if the destination IP isn’t in the lxc_map?”

The answer in that post was basically: the packet gets dropped.

To summarize: The lxc_map is a node-wide map that maintains a mapping from a destination-ip to an EndpointInfo. The EndpointInfo contains an interface index. So, when the desination-ip is known, the eBPF program simply redirects it to that interface.

Every real endpoint (such as a Pod) provisioned by the daemon is in that map. However, a Kubernetes Service ClusterIP is not. A ClusterIP is a virtual address created by the API server; nothing owns it, no Pod answers for it, and the Sarena daemon never puts it in lxc_map.

So what happens when something on the node opens a connection to a ClusterIP?

(note: this post talks about a connection to a ClusterIP whose backend is a pod on the same node. Off-node backends add an overlay hop that is not built yet.)

Reaching a Service ClusterIP: where the datapath stops and the host takes over

The eBPF program from_host does roughly:

fn process_ipv4(ctx: &TcContext, config: &EndpointConfig) -> Res<EbpfReturn> {
    
    let dst_ip = ...;

    if let Some(ep) = lookup_ipv4_endpoint(dst_ip) {
        return local_delivery(ctx, ep);
    };

    Ok(Verdict::Pass)
}

lookup_ipv4_endpoint is a lookup in lxc_map. And since the destination ip is not in the map, the if branch does not run (no local delivery). The program returns Verdict::Pass.

Pass means “I have no opinion, give it back to the kernel.” The packet is handed up into the node’s host network stack as if the eBPF program weren’t there at all.

kube-proxy’s PREROUTING DNAT

Now the packet is a normal packet in a normal Linux network stack, and it goes through netfilter like any other packet. The relevant hook is PREROUTING in the nat table, and the relevant rules are the ones kube-proxy installs and maintains.

kube-proxy watches Service and EndpointSlice objects from the API server and, for every ClusterIP, programs a chain of rules that maps the virtual ClusterIP to one of its ready backend endpoints. In the classic iptables mode, this is implemented as an iptables chain; the other kube-proxy modes use their corresponding mechanisms.

When our packet hits that chain:

1. The chain selects one of the ready backend endpoints. In iptables mode, kube-proxy uses probabilistic rules to distribute new flows across the available endpoints. This is the load balancing step. It happens here, not in Sarena.
2. The selected endpoint is DNATed. The destination is rewritten from the ClusterIP, for example 10.96.0.10:53, to the selected backend, such as 10.0.10.30:53.
3. Conntrack records the translation. The DNAT decision becomes part of the connection’s state, so subsequent packets in the flow use the same translation rather than selecting another backend. The backend is therefore effectively pinned for the lifetime of the connection.

Sarena doesn’t participate in any of this, and it doesn’t need to. Service semantics — such as endpoint selection, session affinity, and exclusion of unready endpoints — are handled by the Kubernetes service proxy. By the time the packet reaches Sarena’s forwarding logic, the destination is a real Pod IP rather than the virtual ClusterIP.

Routing the translated packet

After PREROUTING, the kernel makes its routing decision using the post-DNAT destination. For example, after the translation from 10.96.0.10:53 to 10.0.10.30:53. It looks up 10.0.10.30 in its routing table.

This is where the packet stopped in the previous blog post: The kernel didn’t know how to reach 10.0.10.30, so it dropped the packet.

Host routing

With the new functionality in place, Sarena creates a new veth pair and adds a route for the Pod CIDR during startup.

During startup, Sarena does the following:

let (mut host, _) = setup_host_device(
    ...,
    InterfaceAddress {
        ip: self.config.internal_ip,
        prefix_len: 32,
    },
)
.await?;

if let Some(prefix) = self.config.ipam_ipv4_subnet {
    let route = Route {
        nexthop: Some(self.config.internal_ip),
        local: Some(self.config.internal_ip),
        prefix,
        mtu: Some(1500),
        ..Default::default()
    };
    host.add_route(&route).await?;
}

setup_host_device creates a new veth pair: sarena_host and sarena_net. Both interfaces live in the host network namespace, although sarena_net is not important for this part of the story.

sarena_host is assigned the node’s internal_ip as a /32. Its purpose is to provide a routing target where the datapath can attach an eBPF program.

Then host.add_route(&route) adds a route covering the entire Pod CIDR range (ipam_ipv4_subnet). The nexthop is the node’s internal_ip, which is assigned to sarena_host.

Now the routing decision for 10.0.10.30 succeeds: the destination falls within 10.0.0.0/16, so the kernel routes the packet via internap_ip on sarena_host.

This means that 10.0.10.30 from the previous example no longer gets dropped. Instead, the packet is sent towards sarena_host — where Sarena’s datapath can take over.

Back into the datapath

As the packet is transmitted through sarena_host, the TCX egress hook fires. For the host device, the attached eBPF program is from_host, which has a similar shape to from_container:

if let Some(ep) = lookup_ipv4_endpoint(dst_ip) {
    return local_delivery(ctx, ep);
};

This time dst_ip is 10.0.10.30 — a real pod IP, provisioned by the daemon and present in lxc_map. The lookup hits, and local_delivery redirects to the right interface.

At the time of my previous blog post, the redirect was a simple rewrite of the L2 header. Now, local_delivery does a little more: it decrements the TTL, fixes the IP header checksum, rewrites the Ethernet header so the frame appears to come from the Pod’s gateway and is addressed to the Pod’s MAC, and finally uses bpf_redirect to send the packet directly to the Pod’s host-side interface.

So the ClusterIP indirection is resolved in two independent stages:

1. kube-proxy turns the Service VIP into a real Pod IP during PREROUTING.
2. Sarena’s datapath forwards that Pod IP using the same lookup-and-redirect path it uses for any other Pod destination.

The Pass in the first step is what makes this possible. The datapath has to get out of the way and let the packet continue through the normal Linux network stack so that netfilter — and therefore kube-proxy’s DNAT rules — can run.

So, with all the functionality in place, it’s time for some testing. Theoretically, it should work.

Except… it didn’t.

Watching it in Kubernetes

When I deployed Kubernetes in kind using my Sarena CNI plugin (more on that in a future blog post), I saw that the Pods were provisioned successfully. An IP address is randomly selected from the Pod CIDR range and assigned to each Pod. That all seemed to work.

However, when I watched the Pods starting up, I noticed that CoreDNS never became ready. The error was:

Readiness probe failed: Get "http://10.0.10.30:8181/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The readiness probe is executed by the kubelet on the node and, in this case, needs to connect from the host to CoreDNS at 10.0.10.30:8181.

But the host could not reach 10.0.10.30.

Why?

Following the packet

A kind “node” is just a container, and its image is deliberately minimal: no tcpdump, no bpftool, no ping. Rather than install anything, we can run the host’s own binaries inside the node’s network namespace:

PID=$(docker inspect -f '{{.State.Pid}}' sarena-control-plane)
nsn() { sudo nsenter -t "$PID" -n "$@"; }

nsn ip -o link                                    # find sarena_host and the lxc… veth of CoreDNS
nsn tcpdump -neni sarena_host                     # egress side: does the SYN even appear here?
nsn tcpdump -neni lxcXXXXXXXXXXXX                 # the CoreDNS pod's veth: SYN in, SYN-ACK/RST out?
nsn curl -sv -m3 http://10.0.10.30:8181/ready     # CoreDNS readiness, straight at the pod

The forward path is straightforward.

curl opens a connection from the host to 10.0.10.30. As described in Host routing, the route for the Pod CIDR tells the kernel to send packets for 10.0.10.30 via sarena_host.

And that is one of the purposes of the first tcpdump: it tells us whether the SYN actually reaches sarena_host.

At that point, the from_host TCX program runs on the egress path. It looks up 10.0.10.30 in lxc_map, finds the CoreDNS endpoint, and redirects the packet to its host-side veth, lxcXXXXXXXXXXXX.

The second tcpdump lets us check the next step: did the SYN actually arrive at CoreDNS’s host-side veth?

From there, the packet crosses the veth pair and appears on eth0 inside the CoreDNS Pod. CoreDNS receives the request and sends a SYN-ACK back.

Now we can follow the return path.

The SYN-ACK leaves CoreDNS through eth0, crosses the veth pair, and appears on lxcXXXXXXXXXXXX in the host network namespace. The second tcpdump captures this as well.

At this point, from_container processes the packet. Its destination is the host, not a Pod endpoint, so the lookup in lxc_map misses and from_container returns Pass. The packet is therefore handed back to the normal host network stack.

If everything were working, the reply would continue through the host and eventually reach the socket created by curl.

But that’s not what I observed.

The first tcpdump showed that the SYN reached sarena_host. The second tcpdump showed that the SYN also reached CoreDNS’s veth — and, importantly, that the SYN-ACK came back out of the Pod and arrived on that same veth.

But the SYN-ACK never appeared on sarena_host.

That narrowed the problem down considerably: the Pod received the request and generated a reply, but the reply was being dropped somewhere in the host before it could continue.

Finding the drop

The reply arriving at the host looks like this:

source      = 10.0.10.30
destination = <host>
interface   = lxcXXXXXXXXXXXX

The host kernel therefore sees a packet claiming to come from 10.0.10.30, arriving on lxcXXXXXXXXXXXX.

With strict reverse-path filtering enabled, the host performs a reverse-path check on the packet’s source address. It effectively asks:

If I wanted to reach 10.0.10.30, which interface should I use?

The answer comes from the route we added earlier in Host routing:

10.0.0.0/16 via 10.x.x.x dev sarena_host

So the host expects traffic from 10.0.10.30 to arrive on sarena_host, however, this packet arrived on lxcXXXXXXXXXXXX.

Because strict reverse-path filtering is enabled, the host kernel rejects the packet because the incoming interface (lxcXXXXXXXXXXXX) doesn’t match the interface it would use to reach 10.0.10.30(sarena_host). That is the reason the kernel dropped the packet.

The fix is to disable reverse-path filtering for this datapath.

What this is not, yet

There are still some important limitations to this implementation:

• Off-node backends. If kube-proxy DNATs a ClusterIP to a Pod running on another node, the host’s post-DNAT routing lookup needs a route to that remote Pod IP. The current datapath doesn’t provide one, so there is no path to the remote backend. For now, only same-node backends work.

• Off-node clients. A ClusterIP connection arriving from the node’s physical network interface would enter the datapath through from_netdev, which is currently just a Pass stub. The path described in this post therefore only covers traffic originating on the node itself.

• Other Service semantics. Features such as externalTrafficPolicy, hostPort, NodePort source-IP preservation, and session affinity are implemented through kube-proxy’s rules. Sarena doesn’t need to understand these features, but they haven’t been exercised as part of this datapath yet.

The design goal is that Sarena doesn’t need to know what a Kubernetes Service is. The datapath currently speaks one simple language:

destination IP → endpoint → redirect

A Service is handled entirely by the host network stack. kube-proxy turns the virtual ClusterIP into a real Pod IP, and once that happens, Sarena sees exactly the same thing it sees for any other Pod destination.

That’s the nice part: Services don’t need to become an eBPF feature. They just need to become an IP address before the packet reaches the datapath.

The code is in: https://github.com/erwin-kok/sarena