Building a Crash-Safe eBPF Dataplane Loader in Rust

This post is about a design problem that shows up in many eBPF-based networking project sooner or later, and largely independent of what the dataplane itself actually forwards: how do you attach, track, and tear down kernel-resident programs from a userspace control plane, in a way that survives your own process dying at the worst possible moment?

The project this is drawn from is an early-stage eBPF/XDP/TC dataplane written in Rust on top of aya. The packet-forwarding logic itself — routing, ARP, policy — is still mostly stubbed out. What is built out, and what this post is about, is the layer underneath all of that: the loader that manages the lifecycle of eBPF programs and their pinned kernel state, independent of whatever those programs eventually do.

Implementation: https://github.com/erwin-kok/sarena

Three actions, not one

It’s easy to talk about “installing an eBPF program” as a single action. It’s actually three, and each produces a distinct kernel object:

1. Load — submit a fully-resolved instruction stream to the kernel. The verifier statically proves it’s safe (no invalid memory access, bounded loops, no way to leak kernel pointers) and, if it passes, the kernel hands back a program object.
2. Attach — connect that loaded program to a hook point (a network interface, in this project’s case). This produces a second, distinct kernel object: a link, representing that one specific attachment.
3. Pin — give a kernel object (a program, a link, or a map) a name inside a special filesystem conventionally mounted at /sys/fs/bpf. A pin is nothing more exotic than that: a named, durable reference to an object that already exists.

Separating these explicitly matters because each answers a different question. Loading answers “is this code safe to run.” Attaching answers “where does it run.” Pinning answers something different: “does this still exist after the process that created it is gone” — which is exactly the question the rest of this post is about.

The core problem: kernel state outlives your process

An eBPF program attached to a network interface, once loaded, keeps running whether or not the userspace process that loaded it is still alive. That’s a feature — it’s why eBPF dataplanes can survive a control-plane crash without dropping traffic. But it also means the control plane can never assume it owns a state. On every start, it might be looking at:

• Programs already attached from a previous run.
• Kernel objects (maps, links) pinned to the BPF filesystem, some of them orphaned because the process died between attaching a program and finishing its bookkeeping.
• A “half-attached” endpoint: some of its hooks wired up, others not, because the crash happened somewhere in between.

The pinned files in bpffs (the BPF filesystem, typically mounted at /sys/fs/bpf) are the only durable representation of what’s actually running. Any design that keeps state in a userspace data structure—e.g. a HashMap of endpoints—can become inconsistent after a crash. The architectural decision made early on was: the filesystem is the state. There is no separate userspace state to synchronize; “what’s attached” is defined by what can be discovered by walking the pin tree. Reconciliation, restart recovery, and cleanup all become variations of the same directory walk instead of three separate consistency problems.

This design also makes attach and detach idempotent. Attaching an endpoint that is already attached leaves the kernel state unchanged. Detaching an endpoint after a partial attach removes only the pins that exist. There is no separate recovery logic: normal operation and recovery both reduce to the same operation—make the pin tree match the desired state.

Isolating blocking kernel calls: an actor on its own OS thread

The control plane is otherwise a conventional async Rust binary. It uses tokio, performs netlink operations asynchronously, and assumes that blocking a task is a bug. Loading an eBPF object, attaching a program to a tcx/xdp hook, and pinning kernel objects are different: they are synchronous operations that can take a non-trivial amount of time. The verifier alone can consume significant wall-clock time for a large program. Running these operations on a tokio worker thread would block other tasks scheduled on that worker.

The solution is a dedicated OS thread running a small actor. The actor owns the loader and processes commands from a channel: add an endpoint, remove an endpoint, reconcile the desired set, or list the currently active endpoints. The async side communicates with it through a channel-backed handle and awaits the result. The blocking work is therefore isolated from the tokio runtime; regardless of how long a verifier run or netlink operation takes, it only blocks the actor thread.

The actor’s single-threaded execution model also provides concurrency control. Requests cannot race against each other while modifying the pin tree, because they are serialized before they reach the loader. An add and a remove for the same endpoint execute in a defined order rather than interleaving. This avoids a class of synchronization problems—such as concurrent pin/unpin races—without requiring locks.

Making kernel access testable without a kernel

eBPF code and its loader cannot be exercised in a normal unit test environment. The kernel execution context is unavailable, and the userspace side requires privileges and resources such as CAP_NET_ADMIN and a writable bpffs.

The solution is to make the loader boundary explicit. The control plane depends on a trait that can load programs, attach hooks, and manage pins. Production uses the real loader and syscalls; tests use an in-memory implementation that models the pin tree and can inject failures at arbitrary points in a sequence.

This makes it possible to test the failure cases that matter: a crash after loading an object, after one attachment, or halfway through cleanup. Tests can assert the resulting pin-tree state, restart from that state, and verify that reconciliation produces the same result as a clean run. The logic is tested without root, namespaces, or a running kernel.

Real kernel integration tests still exist, but they cover only the parts that require the kernel: namespace isolation, veth setup, and actual attach/detach operations.

A loaded program’s data bindings are permanent

A loaded eBPF program is permanently bound to the maps it was loaded with. The object file declares the maps it uses; loading resolves those declarations into concrete kernel objects and writes their identifiers into the program instructions before verification. The verifier proves that exact instruction stream satisfies the kernel’s safety requirements. There is no later operation that rebinds a program to different maps.

The consequence is that loading the same program twice against different maps creates two independent program objects, each with its own fixed bindings. There is no shared program that can be redirected between configurations at runtime.

This constraint determines how multiple logical instances—containers, endpoints, tunnels—can share program logic while keeping their data separate:

• Load per instance, rename the maps. Each instance gets its own program load and its own uniquely named kernel objects. This provides full isolation and simple cleanup, at the cost of running the verifier once per instance.

• Map-of-maps. Load the program once and use an outer map to select the per-instance inner map at runtime. This removes load-time cost for new instances, but adds an extra lookup and introduces a shared indirection table that userspace must maintain correctly.

• One shared map, keyed by instance. Store all instances in one map and include the instance identifier in the key. This avoids per-instance program loading and keeps lookup costs low, but all instances share the same map capacity and lifecycle. One noisy instance can consume resources needed by others.

This project uses the first approach: each endpoint gets its own loaded program with endpoint-specific map bindings. At the current scale, the verifier cost is paid once per endpoint lifecycle, while the isolation and simple cleanup model are worth more than avoiding those loads. If endpoint churn becomes the bottleneck later, map-of-maps is the natural evolution path; the program logic remains the same, only the data binding model changes.

Endpoint identity: resisting the urge to invent an ID

Renaming maps per instance raises a simple question: renamed to what? The first version introduced a synthetic numeric identifier for one endpoint type, based on an assumed kernel naming constraint. Once that constraint was verified to apply elsewhere, the extra identifier had no purpose and was removed. The network interface name itself was already the correct identity: it was provided by the kernel, already unique in the required scope, and already bounded by the kernel’s own limits.

The broader rule is that an identity scheme should contain only the structure the problem requires. Every additional identifier creates another piece of state that must be allocated, propagated, and kept consistent. If an existing identifier already has the right semantics, introducing another one only creates another consistency boundary.

The same principle applies to the rest of the design: pin paths are derived from (endpoint kind, interface name), and discovery reconstructs state by walking that structure rather than consulting a separate index. One source of truth, derived consistently.

Pinning as a place where kernel behavior gets subtle

Some of the harder bugs came not from userspace design, but from incorrect assumptions about kernel and loader behavior.

Kernel pinning is declared on the object itself, not inferred from the loader call. A map or link without pinning enabled will simply not appear in bpffs, even if the loader provides a pin path. The fix was straightforward once identified: explicitly declare pinning for every object that must survive restart.

Stale pins are another example. A pinned link can outlive the network device it was attached to. Before reusing the same pin path, the control plane must detect and remove that stale object; otherwise the new attach fails because the path is already occupied.

The common lesson is that kernel state transitions are part of the design. The important tests are not only the happy path, but the transitions around it: crash after step N, device removal with a surviving pin, and recovery from partial state.

Observability inherits the same constraint

The load-time binding rule applies to observability as well as data maps. eBPF logging (aya-log) uses a ring-buffer map, and that map is bound when the program is loaded. Because this project loads programs per endpoint, each endpoint gets its own log stream as a consequence of the same isolation model.

The log stream must therefore follow the endpoint lifecycle. The userspace handle that drains the ring buffer must remain alive; otherwise the endpoint’s log stream has no reader, so log draining starts when an endpoint loads and stops when that endpoint is removed. There is no global logging setup; logging is part of the endpoint state machine.

The same boundary appears in two smaller places. The eBPF logging layer uses a simpler logging interface than the control plane, so a small adapter bridges the two rather than forcing either side to change. And because ring buffers require active draining, the existing dedicated-thread pattern is reused: one blocking reader per endpoint, isolated from the async runtime and released with the endpoint.

Log records also do not contain endpoint identity. That information exists only in the userspace context that owns the stream, so the identity is attached at the draining boundary rather than threaded through every log call. The result is the same pattern used elsewhere in the design: keep context at the boundary where it naturally exists, instead of duplicating it through the system.

None of this was a separate logging design decision. It is the consequence of applying the same per-instance loading model consistently.

Where this leaves things

None of this describes packet forwarding itself; that part of the project is still early. The point is that lifecycle management underneath a dataplane is its own engineering problem, with its own failure modes: crash recovery, partial state, stale kernel objects, and testing across a kernel boundary.

The result is a lifecycle layer with explicit ownership of: durable state owned by the kernel, blocking operations isolated from async execution, a testable boundary around kernel access, and identity and data models that introduce no unnecessary state. With those pieces in place, the forwarding logic can be built on top of a system whose state transitions are explicit and recoverable.

The important design choices are not isolated to one subsystem. A decision such as per-instance loading affects data ownership, cleanup, and even observability. The consequences follow wherever the same constraints appear.