Writing an XDP program that loads and passes traffic correctly is the easy part. Writing it in such a way that it will actually deliver packet rate is a different kind of work. Almost none of the levers involced inside your .c file. Most of the throughput lost to attach mode choice, queue/IRQ layout, map access patterns, and NUMA placement, not to anything the verifier would ever flag.

This post is like a checklist of all these things that can actually move the needle on your side.

Confirm You’re actually in Native Mode

My blog XDP Hook Internals — Where You Actually Intercept Packets covered this in depth, so briefly: everything below assumes that you are in native(driver) mode of XDP. If your program silently falls back to generic mode, none of the rest of this post will get you anywhere close to the numbers you’re expecting, because you’ve already paid the SKB-allocation cost, for which XDP exists to avoid.

The published, reproducible numbers for native-mode XDP_DROP comes from the original XDP paper’s own benchmark suite: ~40 ns/packet(single core, mlx5). That’s the ceiling this whole post is chasing a fraction of — check it before tunning anything else:

ip -d link show eth0

# xdp/id:12 <- native, good
# xdpgeneric/id:12 <- generic, stop and fix this first

Queue Count and IRQ Affinity, Not Core Count

XDP has no schedular. A packet arriving at RX queue N runs your program on whatever CPU that queue’s IRQ is currently affinitized to — there’s no cross-CPU dispatch, no rebalancing, nothing. If your NIC has 8 queues and 6 of them share IRQ affinity with CPU 0, you have effectively 3 cores of XDP throughput no matter how many cores the box has.

ethtool -l eth0 # how many queues are configured
cat /proc/interrupts | grep eth0 # which CPU each queue's IRQ landed on
cat /sys/class/net/eth0/device/numa_node

Two independent things to get right, and both are configuration, not code:

  • Queue count matches core count you intend to use. ethtool -L eth0 combined N sets it; more queues than cores you’re willing to dedicate just mean some queues share a CPU.
  • IRQ affinity is spread, not clustered. irqbalance running alongside a latency-sensitive XDP deploymenet is common self-inflicted wound — it actively moves IRQs around at runtime, which is exactly what you don’t want when you’ve deliberately pinned queue-to-CPU mappings. Disable it on XDP boxes and set /proc/irq/N/smp_affinity explicitly.

This is the single highest-leverage thing on this list, and it’s also the one people skip because it’s not eBPF code — it’s ethtool and /proc .

Keep the Program on the NUMA node the NIC is on

On any multi-socket box, check which NUMA node actually owns the NIC before you decide which cores get the XDP-handling IRQs:

cat /sys/class/net/eth0/device/numa_node
lscpu | grep NUMA

If that comes back 1 and your IRQs are pinned to node 0’s cores, every packet touch — reading the descriptor, running your program, writing back a verdict — crosses the inter-socket interconnect. This is standard, vendor-documented guidance (Mellanox’s own performance tuning docs say this explicitly for their adapters), not an XDP-specific quirk — it’s the same rule as any other high-packet-rate workload. I’m not aware of a clean, universally-quotable “X% penalty” number for this, because it depends on the specific interconnect and workload; treat it as “keep this aligned, always” rather than something you need to benchmark to justify.

Pick the Map Type for the Access Pattern, Not the Data Shape

The map is usually the only per-packet memory access in an XDP program that isn’t the packet buffer itself, so its cost matters disproportionately.

  • BPF_MAP_TYPE_ARRAY: for anything keyed by a small, dense integer (a queue index, a fixed set of counters, a config slot) — it’s a direct index into a backing array, no hashing, no bucket walk. If your key space is 0..N and small, this is the fastest map type you can reach for.
  • BPF_MAP_TYPE_HASH: for sparse or unpredictable keys. Correct default for most lookups, but it’s doing real work per access: computing a hash, walking a bucket.
  • BPF_MAP_TYPE_LPM_TRIE: for CIDR/prefix matching — and it is measurably slower than an exact-match hash lookup, because the cost scales with trie depth. Cloudflare’s published LPM trie benchmarks show throughput dropping from ~7.4M ops/sec at 10K entries to ~1.5M ops/sec at 1M entries on the same hardware, specifically because more entries means a deeper trie to walk. If you’re doing exact-match IP lookups (not prefix matching), a plain hash map keyed on the IP is faster — reach for LPM_TRIE only when you actually need longest-prefix-match semantics, not as a default “IP key” map type.
  • BPF_MAP_TYPE_PERCPU_* variants : whenever the value is written frequently and doesn’t need cross-CPU visibility (counters, per-flow scratch state) — covered in depth in the maps and tail-calls posts earlier in this series. The short version: it turns a contended access into an uncontended one, at the cost of needing to aggregate across CPUs from userspace when you actually read it.

None of this is about correctness — every map type gives you the same result. It’s purely about how much work happens between “packet arrives” and “verdict returned,” multiplied by every packet you process.

Don’t Touch bpf_xdp_adjust_head/bpf_xdp_adjust_meta More Than Necessary

Both helpers are real and exist for legitimate reasons — bpf_xdp_adjust_head() grows or shrinks headroom (for encapsulation/decapsulation), and bpf_xdp_adjust_meta() extends the data_meta region so a later tc-BPF program on the same device can read metadata your XDP program computed. There’s no separate published benchmark for “cost per call” that I’d point to, but the underlying reason to avoid calling them speculatively is the general principle behind this whole post: every helper call is per-packet work, and both of these can involve moving data or metadata, not just adjusting an offset. If a program calls adjust_head on a code path that turns out not to need the extra headroom, that’s pure waste. Compute whether you need the adjustment first — the branch is nearly free; the unconditional adjustment is not.

Batch Redirects Are Already Batched — Don’t Fight It

If you’re using XDP_REDIRECT (to a DEVMAP , CPUMAP , XSKMAP ), the kernel is already doing the batching for you: bpf_redirect_map() doesn’t move the packet immediately, it queues it, and the driver calls xdp_do_flush() once at the end of each NAPI poll to flush everything queued during that batch. This is confirmed directly in the kernel source ( kernel/bpf/devmap.c , kernel/bpf/cpumap.c ) and in the kernel’s own BPF redirect documentation.

The practical implication: there’s no per-packet syscall or per-packet flush to optimize away yourself — that work is already amortized over the whole NAPI batch. What you can get wrong is calling bpf_redirect_map() more than once for the same packet, or on a path that also returns a non- XDP_REDIRECT verdict — the batching helps you, but only if your program’s control flow is a clean “redirect XOR something else” per packet.

CPUMAP redirect specifically exists for software load rebalancing — moving a packet to a different CPU’s processing queue when the NIC’s own RSS hashing is landing too much traffic on one queue’s CPU. The redirected CPU does the SKB allocation and normal stack processing itself; this is documented as an explicit RSS-rebalancing use case in the kernel’s cpumap docs, not just a side effect.

XDP_TX: You Own the Checksum Now

Covered already in Part 4, worth repeating here because it’s a performance bug as much as a correctness one: after XDP_TX , nothing recomputes checksums for you. If your program modified any field covered by a checksum — rewritten IP for DSR, a NAT’d port — and you don’t call bpf_l3_csum_replace() / bpf_l4_csum_replace() (or manage the diff yourself with bpf_csum_diff() ), you’re not looking at a performance problem, you’re shipping corrupt packets. This only makes this list because the “fix” people sometimes reach for — recomputing the whole checksum from scratch in a loop over the packet — is far more expensive than the incremental update these helpers are built to do in one call.

Keep the Verifier’s Job Boring

Tail Calls and Program Chaining — Building Large eBPF Applications already established the concrete numbers: the verifier’s real budget is BPF_COMPLEXITY_LIMIT_INSNS , 1 million processed instructions, and every branch on an unknown value forks the register state the verifier has to track — a 548-instruction program hit that ceiling in the measurement from that post, while a 16,388-instruction straight-line program didn’t come close. That’s not a load-time concern only. A program with many branches and many map lookups per packet is doing more runtime work per packet too — the verifier complexity and the actual per-packet cost tend to move together, because both are driven by the same thing: how much the program does before it returns a verdict.

The pattern that post already covered — pushing genuinely optional, branchy logic (a DNS inspector, a rare protocol path) behind a tail call so the common case stays a short, straight-line program — is worth restating here as a performance technique, not just a verifier-budget one. Most packets should take the cheapest path through your program; anything conditional and expensive belongs in a stage most packets never reach.

Measure With Tools That Don’t Lie to You

iperf3 and similar bandwidth tools tell you almost nothing about packet-processing cost — a handful of large segments can saturate a link while exercising your XDP program a few hundred times a second. Two tools that actually stress the packet-rate path:

  • pktgen ( net/core/pktgen.c , in-tree) — generates line-rate synthetic traffic from the kernel itself, no userspace packet-copy overhead skewing the result.
  • xdp-bench (part of xdp-tools ) — purpose-built for exactly this: drop/pass/tx/redirect benchmarking against a real loaded XDP program, reporting packets/sec per core.

Whatever you use, sample by packet count, not wall-clock time, and report a distribution across multiple runs rather than a single number — cache state, IRQ scheduling jitter, and background load on the box all move the result more than people expect on a first run.

References

More on eBPF internals: