Basic eBPF forwarding

This post is about (very) basic pod-to-pod networking using eBPF. The goal is to create two endpoints, wire them together into an eBPF dataplane, and then send a UDP packet from one to the other to prove that the whole path works using the bare minimum.

The scope is deliberately narrow: same-L2-segment, same-host pod-to-pod forwarding. There is no overlay, cross-host routing, IPv6, policy, conntrack, or upstream connectivity. Those are future work. For now, the goal is to build the smallest possible dataplane that can get one packet from one pod to another.

The topology

Each pod gets a veth pair. One end is moved into the pod’s network namespace (the “peer” end); the other stays in the host’s namespace (the “host” end).

This is completely ordinary Linux networking. The interesting part is what we configure on each end — and, just as importantly, what we don’t configure.

• Peer end (inside the pod): gets a locally administered MAC address and a pod IP address, for example 192.168.1.10/32.
• Host end (in the daemon’s namespace): gets only a MAC address. No IP address is configured on it.

The host end is where the eBPF programs are attached. No eBPF programs are attached to the peer side. The host end is also the interface that the dataplane redirects packets to.

It does not need an IP address because it does not participate in host IP routing. It needs a MAC address because it represents the virtual gateway from the pod’s point of view and is used when answering ARP requests for that gateway.

A gateway that does not exist

Every pod gets exactly two routes pushed into its namespace:

1. A /32 host route to a fixed gateway address (10.0.0.5 in the test), marked on-link: “this address is directly reachable.”
2. A default route (0.0.0.0/0) with that same address as its next hop.

The interesting part is that 10.0.0.5 isn’t assigned to anything.

It’s not the host end of the veth, it’s not any real interface on the machine.
As far as the pod’s network namespace is concerned, 10.0.0.5 is simply the gateway it should use for traffic outside its own local network.

This still uses the kernel’s ordinary routing logic. When the pod wants to send a packet to 192.168.2.20, for example, the default route says that the next hop is 10.0.0.5. Because the gateway route is marked on-link, the kernel needs to resolve the gateway’s MAC address with ARP.

And that’s where the eBPF dataplane takes over.

ARP for a gateway that isn’t there

The first time a pod needs to send anything, it ARPs for 10.0.0.5. That request leaves the pod’s peer interface, arrives at the host end of the veth.

At that point it is intercepted by the from_container eBPF program, which processes the ARP packets before it ever reaches normal kernel ARP handling.

pub fn from_container(ctx: TcContext) -> Res<EbpfReturn> {
    let ethhdr: *const EthHdr = unsafe { ptr_at(&ctx, 0)? };

    let Ok(ether_type) = unsafe { *ethhdr }.ether_type() else {
        return Ok(EbpfReturn::Pass);
    };

    let config = ENDPOINT_CONFIG
        .get(0)
        .ok_or(InternalError("endpoint does not have EndpointConfig"))?;

    let result = match ether_type {
        EtherType::Ipv4 => process_ipv4(&ctx, config)?,
        EtherType::Arp => process_arp(&ctx, config)?,
        _ => EbpfReturn::Pass,
    };
    Ok(result)
}

The program checks what the ARP request is asking for. Requests for the gateway address are handled by eBPF. Requests for the pod’s own address, such as gratuitous or self ARP, are allowed through unchanged.

For a gateway request, the program constructs an ARP reply in place:

1. Swap the Ethernet source and destination MAC addresses.
2. Swap the ARP sender and target addresses.
3. Change the ARP operation from request to reply.
4. Put the configured gateway MAC into the reply.
5. Redirect the packet back to the pod with bpf_redirect_peer().

pub fn process_arp(ctx: &TcContext, config: &EndpointConfig) -> Res<EbpfReturn> {
    let ethhdr: *const EthHdr = unsafe { ptr_at(&ctx, 0)? };
    let arphdr: *const ArpHdr = unsafe { ptr_at(&ctx, EthHdr::LEN)? };

    if !arp_matches(&ctx, ethhdr, arphdr, &config.mac) {
        return Ok(EbpfReturn::Pass);
    }

    let eth = unsafe { &*ethhdr };
    let arp = unsafe { &*arphdr };
    let smac = eth.src_addr;
    let spa = arp.spa();
    let tpa = arp.tpa();

    if tpa == config.ipv4.octets() {
        return Ok(EbpfReturn::Next);
    }

    info!(
        &ctx,
        "arp: who-has {:i}? (default) replying with {:mac}", tpa, config.mac
    );

    let eth_mut = ethhdr as *mut EthHdr;
    let arp_mut = arphdr as *mut ArpHdr;
    arp_prepare_response(eth_mut, arp_mut, config.mac, tpa, smac, spa);

    let ifindex = ctx_get_ifindex(ctx);
    let ret = ctx_redirect_peer(ifindex, 0);
    Ok(EbpfReturn::Custom(ret as i32))
}

The ARP request therefore never leaves the veth pair. From the pod’s point of view, it received an ordinary ARP reply from 10.0.0.5.

Nothing on the host side needed an IP address to make this work.

There is another important detail here: each pod has its IP configured as a /32. This means the pod has no connected IPv4 subnet containing the other pods.

Consequently, the pod does not ARP for another pod’s IP address. Instead, its routing table sends non-local traffic to the virtual gateway, and the gateway is the only next-hop address it needs to resolve with ARP.

There is therefore no need for ARP to be flooded between the pod interfaces. The ARP request arrives at the host-side veth and is handled directly by the eBPF program, which answers on behalf of the gateway. From the pod’s perspective, the gateway answers every ARP request that matters (excluding self-ARP), regardless of which actual endpoint the eventual packet is destined for.

Sending the packet: the actual forwarding step

Once ARP has completed, the pod knows the MAC address of its “gateway” and can send the UDP packet.

Suppose pod A (192.168.1.10) sends a packet to pod B (192.168.2.20). The packet looks roughly like this when it reaches the host-side veth:

Ethernet
  dst MAC = virtual gateway MAC
  src MAC = pod A MAC

IPv4
  src IP = 192.168.1.10
  dst IP = 192.168.2.20

UDP
  ...

That packet also arrives at the host end of the veth and hits from_container — this time going through process_ipv4 instead of process_arp.

fn process_ipv4(ctx: &TcContext, config: &EndpointConfig) -> Res<EbpfReturn> {
    let ethhdr: *const EthHdr = unsafe { ptr_at(&ctx, 0)? };
    let ipv4hdr: *const Ipv4Hdr = unsafe { ptr_at(&ctx, EthHdr::LEN)? };

    let eth = unsafe { &*ethhdr };
    let ipv4 = unsafe { &*ipv4hdr };

    if let Some(ep) = lookup_ipv4_endpoint(dst_ip) {
        let ifindex = unsafe { (*ep).if_index };
        let dst_mac = unsafe { (*ep).mac };

        let eth_mut = ethhdr as *mut EthHdr;
        unsafe {
            (*eth_mut).dst_addr = dst_mac;
            (*eth_mut).src_addr = config.mac;
        }

        let ret = unsafe { bpf_redirect(ifindex, 0) };
        return Ok(EbpfReturn::Custom(ret as i32));
    }

    Ok(EbpfReturn::Pass)
}

The forwarding decision doesn’t care about the destination MAC at all (it’s just the MAC of the gateway) — it looks at the packet’s destination IP.

Every endpoint’s information is recorded in one shared, node-wide map, lxc_map, keyed by IP address:

lxc_map: Ipv4Addr → EndpointInfo { if_index, mac }

if_index is the index of the destination pod’s host-side veth interface. This must be the host-side interface because the eBPF program is running in the host namespace, and interface indexes are meaningful within a network namespace. All of the host-side veth interfaces live in the same namespace, so the program can redirect between them.

The mac is the MAC address of the destination pod’s peer-side interface. This is needed because the packet is ultimately delivered to the pod’s peer interface. From the pod’s perspective, the incoming Ethernet frame must have its own MAC address as the destination.

Whenever the daemon creates an endpoint, it inserts the pod’s IP into this map pointing at its own host-side ifindex and its peer-side mac address:

let mut lxc_map: HashMap<_, Ipv4Key, EndpointInfo> = HashMap::try_from(map)?;
let key = Ipv4Key::from_addr(peer_ip);
let value = EndpointInfo {
    if_index: link.ifindex(),
    mac: peer_mac.0,
};
lxc_map.insert(key, value, 0)?;

So the flow for a IP packet from pod A to pod B is:

1. The packet leaves pod A’s peer interface and arrives at pod A’s host-side veth.
2. from_container looks up the destination IP in lxc_map and finds pod B’s host-side ifindex and peer-side mac address.
3. It changes the destination MAC to pod B’s peer-side MAC.
4. It changes the source MAC to the statically configured MAC of the host-side endpoint.
5. It calls bpf_redirect with pod B’s host-side ifindex.
6. The packet comes out on pod B’s host-side veth end, crosses the pair, and arrives at pod B’s peer interface as if it had just come in off the wire.

That’s the entire forwarding path.

There is no Linux IP route on the host, no bridge connecting the host-side interfaces, and no central forwarding process. The eBPF program uses the destination IP to find the destination interface and MAC address, rewrites the Ethernet header, and redirects the packet directly to that interface.

When the destination isn’t known

What happens if the destination IP isn’t in lxc_map?

There is no redirect target, so the eBPF program returns Pass rather than actively redirecting the packet.

Because the host has IP forwarding disabled and there is no normal host route that would take the packet anywhere useful, the packet effectively goes nowhere.

Functionally, this behaves like a drop. It is not yet an explicit, intentional “drop because this destination is unknown” dataplane decision. Routing to anything outside the set of locally known endpoints — another node, the outside world, a service IP, and so on — isn’t implemented yet.

What this test setup does not cover

This scenario intentionally leaves out a lot.

• IPv6. Everything is v4-only — the peer interfaces only ever get a v4 address, and the eBPF dispatch only special-cases Ipv4/Arp IPv6 traffic just falls through as Pass.
• Cross-host forwarding. Both pods live on the same host. There is no overlay or tunnel involved.
• Anything reaching a real upstream. to_container (traffic heading into a pod from elsewhere) and the host/netdev-facing programs are currently pass-through stubs — this scenario never needs them, since both ends of the conversation are pods the daemon already knows about.
• Policies and conntrack. There are none. Every known IPv4 endpoint is reachable.

None of that is a limitation of the approach — it’s just not built yet. This test is deliberately the smallest possible slice: two pods, one host, proving that ARP spoofing for a virtual gateway plus an IP-keyed redirect table is enough to get a packet from one pod to another.

The code is in: https://github.com/erwin-kok/sarena

The test setup discussed here is in: sarena-daemon/src/main.rs