Home Posts Tags About

Contents

Testing eBPF code where it actually runs

     1975 words   9 minutes 

This framework is heavily inspired by Cilium’s eBPF testing approach. The overall testing architecture comes from Cilium; this post focuses on implementing those ideas in Rust using Aya. If you’re interested in the original design, I recommend taking a look at Cilium’s testing framework. The implementation described in this post is available in my GitHub repository.

When exploring/implementing an eBPF/XDP/TCX dataplane, a question arose: “How do we actually test this?” For example, when we have an eBPF program called process_ipv4(), how can we test its functionality? Granted, these programs are written in Rust (when using Aya), so it looks straightforward to test them like any other Rust function. However, there are a few caveats:

  • In production, process_ipv4() runs in kernel-space, not in user-space, therefore it must be verifier-friendly. We cannot just write any code: it must be accepted by the verifier.
  • Testing process_ipv4() in userspace doesn’t necessarily tell us how it behaves once it has passed the verifier and is executing in the kernel.
  • eBPF has specific intrinsics — special maps, tail-calls, special bpf_* calls, etc. — emulating these is not trivial, and it’s not testing the actual kernel execution path.

This makes ordinary Rust testing awkward, and the most reliable approach is therefore to execute the verifier-approved program in the kernel itself. Fortunately, Linux provides exactly such a mechanism: bpf_prog_test_run, which executes an already-loaded eBPF program against supplied packet data.

This approach looks trivial: set up everything, call the eBPF program in kernel-space using bpf_prog_test_run, and then check whether the results match the expected outcome. However, it’s not that trivial: we cannot just set up a network packet in userspace, hand it over to kernel-space, and let the eBPF program process this handcrafted packet. When calling bpf_prog_test_run, it needs two important parameters: data (the actual packet bytes) and a context (the metadata, for example where the packet begins and ends).

While userspace can provide packet bytes, it cannot faithfully construct the execution context expected by an eBPF program. Fields such as packet boundaries ultimately refer to kernel memory, and the kernel may rewrite parts of the context before execution.

Therefore, not only does the program itself, like process_ipv4(), run in kernel-space — the setup and checks run in kernel-space too, as eBPF programs, following the classic Arrange / Act / Assert shape:

  • arrange — builds an input packet in the test buffer.
  • act — runs the production code under test against that packet.
  • assert — inspects the resulting packet/context and records pass/fail verdicts — from inside the kernel, since that’s the only place that can inspect the data and context properly.

Only the orchestration (loading the .o files, wiring up test runs, initializing maps, printing results, etc.) happens in userspace.

Authoring a test: #[arrange] / #[act] / #[assert]

Because every test consists of three programs, we need a way to group them — i.e., to know which three programs together make up one test. The name of each program follows a special structure that encodes both the phase and the test name:

__test_fw_{ptype}_{name}

The prefix __test_fw_ denotes that this is a special test framework function. {ptype} denotes the phase of the function — one of arrange, act, or assert — and {name} is the name of the test, which must be identical across all three parts.

To help write these functions, three macros are defined: #[arrange], #[act], and #[assert]. These macros rename the function accordingly. They also accept either tc or xdp as a parameter, which denotes the section these programs are placed in.

arrange and act functions have the same shape: they take a context and return TestStatus. assert functions are a bit special (see the next section): they take a context and a TestSuite as parameters, and return nothing.

As an example:

#[arrange(tc, "basic_test")]
pub fn does_not_matter_but_unique_1(ctx: TcContext) -> TestStatus {
    // setup packet
    // ...
    
    TestStatus::Pass
}

#[assert(tc, "basic_test")]
pub fn does_not_matter_but_unique_2(ctx: TcContext, t: &mut TestSuite) {
    let data = ...;
    let data_end = ...;
    let len = ...;

    // check/assert status
    assert_test!(
        t,
        data + len <= data_end,
        "ctx too short: need %d bytes",
        len
    );
}

As mentioned, functions annotated with arrange, act, or assert are eBPF programs too. This means we’re in no_std territory: no allocator, no format!, no Display. However, when writing a test we’d still like some checks, like in regular Rust:

assert_eq!(a, b, "a should be equal to b ({a} == {b})");

That’s not possible in an eBPF program. To get some (limited) asserting/logging functionality anyway, the test still evaluates its conditions in-kernel — that part doesn’t change — but instead of printing or panicking, it records the result — name, file, pass/fail status, log lines — into an eBPF map. The runner then reads that map and turns it into a human-readable report: formatting the log messages and printing the verdict the kernel-side code already decided.

Concretely, that all has to be serialized into a fixed-size byte array living in a BPF map (test_suite_result, 8 KiB), because that’s the only communication channel back to userspace. no_std + without an allocator, dynamic containers such as Vec are unavailable, and critically no format! or Display — the verifier doesn’t run an allocator or a formatting machine, so anything resembling Rust’s string formatting is off the table in kernel space.

That constraint is why logging is deliberately primitive: test_log!/assert_test!’s format strings are static, printf-style literals with up to a handful of u64 arguments, not format!().

The framework doesn’t interpolate anything in the kernel — it writes the literal format string bytes and the raw u64 argument values as separate TLV fields, tagged LogFmt / LogArg. All actual substitution (%llu, %d, %x, %p, …) happens later, in userspace, in report::format_log. This is also why arbitrary Debug/Display formatting of dynamic data — e.g. “print this struct” — isn’t supported: if it isn’t a u64 you can smuggle out as a LogArg, it can’t be logged.

TestSuite is responsible for holding the TlvWriter and the test’s status. Several convenience macros exist for logging and asserting.

As an example:

#[assert(tc, "1_basic_test_equal_bytes")]
pub fn basic_test_equal_bytes_assert(ctx: TcContext, t: &mut TestSuite) {
    ...
    test_log!(t, "expected drop, got forward, flags=%llu", flags as u64);
}

Here, a &mut TestSuite is provided to the assert function as t, and using t we can log. In this case the format string "expected drop, got forward, flags=%llu" is written to the TLV buffer, together with its argument, the value of flags. Userspace retrieves the format string and its arguments, and does the actual substitution there.

A few other macros are available as well:

  • assert_test!(t, cond) / assert_test!(t, cond, "fmt", args...) — the general-purpose assertion; if cond evaluates to false, logs and fails the test.
  • assert_buffer!(t, &ctx, name, first_layer, offset, buf, len) — compares packet bytes; discussed further down, under scapy diffing.
  • test_log!(t, "fmt", args...) — printf-style logging.
  • test_fatal! / test_skip! — fail-and-break / skip-and-break out of the macro-injected loop { ... break; } wrapping the function body.

Note that the #[assert] macro wraps the annotated function’s body in a loop — that’s what lets test_fatal!/test_skip! exit early: calling break exits the loop, completing the test as failed or skipped.

Skipping a test is convenient, for example, when a check depends on a feature flag:

if !feature_enabled() {
    test_skip!(t);
}

Execution runs the test body up to this check; if the feature isn’t enabled, the rest of the test is skipped. This check is usually placed at the start of a test.

All of this — every arrange/act/assert group, across every test file — runs inside a single, ordinary Rust test: ebpf_test_runner, a plain #[test].

It starts by loading two compiled objects: the production .o (dataplane-ebpf-programs.o) and the test .o (dataplane-ebpf-test-programs.o).

The runner populates the test object’s tail-call map with the production classifier’s file descriptor, so that an act program can tail-call into the real production code path — the system under test is the actual compiled artifact, not a stand-in.

Next, it scans every program name in the test .o for the __test_fw_{ptype}_{name} pattern from earlier and groups them by {name} into arrange/act/assert triples — arrange and act are both optional, only assert is mandatory. This is the entire test-discovery mechanism; there’s no separate registry. Get the naming wrong and a test doesn’t fail — it’s simply never run.

Then, one test group at a time, the runner:

  1. Resets that test’s maps (test_suite_result, scapy_assert_map_count, and so on), so a previous test’s results can’t leak into this one.
  2. Sets up a zeroed data/ctx buffer pair and runs arrange against them via bpf_prog_test_run.
  3. Feeds arrange’s output — the packet bytes and context it built, both opaque to the runner — into act as input, and act’s output into assert the same way.
  4. Reads the TestResult (name, file, status, log lines) back out of test_suite_result and hands it to report.rs, which prints a colored [PASS]/[FAIL]/[SKIP]/[INTERNAL ERROR] line plus any log messages.

A test comes back in one of four states:

  • Pass — the test succeeded.
  • Fail — an assertion failed.
  • Skip — the test was intentionally skipped, e.g. via test_skip! (see above).
  • FrameworkError — something went wrong in the framework itself rather than in the thing being tested, e.g. the result buffer filled up or a map lookup failed.

If the verdict is Fail or FrameworkError, the runner panics — which is what makes cargo test/just ebpf-test report the whole thing red. Running the suite looks like this:

===== RUNNING eBPF TESTS =====


[PASS] 1_basic_test_equal_bytes

[PASS] 2_basic_test_non_equal_bytes
ebpf/dataplane-ebpf-test-programs/src/scapy_tests.rs:41:
CTX and buffer '&scapy_bytes::SCAPY_SST_REP_BYTES' content mismatch


[PASS] 3_basic_test_too_short
ebpf/dataplane-ebpf-test-programs/src/scapy_tests.rs:102:
CTX len (42) - offset (0) < LEN (50)


[PASS] 4_large_packets

[PASS] 5_xlarge_packets

[SKIP] l2_announcement_arp_no_entry
ebpf/dataplane-ebpf-test-programs/src/arp.rs:28:
expected drop, got forward, flags=4


test ebpf_test_runner::ebpf_test_runner ... ok

Every group runs inside that one #[test], which is a real practical constraint: there’s no way to run a single eBPF test in isolation with cargo test some_name. Narrowing scope means editing which #[arrange]/#[act]/#[assert] functions exist in the source, not filtering by test name.

  • No dynamic string formatting. Log/assert messages are static string literals with u64 arguments substituted on user-space side; there’s no Display/Debug/format! equivalent available in no_std eBPF, so you can’t log arbitrary structured data — only what fits through a handful of integer argument slots.
  • One process, one #[test]. All eBPF tests execute inside a single Rust #[test], meaning individual tests cannot currently be selected via cargo test. A build.rs-generated one-#[test]-per-group split is a plausible future direction, untried so far.
  • Naming convention is the entire discovery mechanism. The regex ^__test_fw_(arrange|act|assert)_(.+)$ against program names is test discovery. A program that doesn’t match this shape is simply never run — silently, with no error — which makes the macro-generated naming scheme essential.
  • Pre-installed .o files required. Because this exercises real bpf_prog_test_run against loaded programs, these programs (production and test) must be available in the EBPF_DIR first.
  • Root - bpf_prog_test_run and loading eBPF programs into the kernel requires certain privileges. The mentioned test_runner is excluded when cargo test is run, because this does not have elevated privileges. The eBPF tests are run before a sudo call is made (just ebpf-test).

The alternative to all of this machinery is testing eBPF/XDP/TC logic indirectly — unit-testing helper functions that happen to be #[cfg]’d to also compile on the host, or trusting integration tests that only run against a live network namespace. Both leave the actual verifier-checked, kernel-executed bytecode path unverified. By making the test’s Arrange, Act, and Assert phases themselves compile to eBPF and execute via bpf_prog_test_run, this framework tests the dataplane the way it will actually run: as BPF bytecode, in the kernel, against the real verifier — with just enough userspace orchestration to load .o files, wire up tail-calls, and turn a TLV byte buffer back into a readable pass/fail report.

Updated on 2026-07-19
 | Home