eBPF XDP Load Balancing Cuts Kubernetes Tail Latency 40%

Networking

Date:November 2, 2026

Topic:

eBPF XDP Load Balancing Cuts Kubernetes Tail Latency 40%

⏱ 2 min read

Your Kubernetes cluster runs fine at 2 AM. At 2 PM, p99 latency spikes 300%. The culprit isn't your application code—it's kube-proxy iptables chains choking on connection tracking under load. We replaced that path with eBPF XDP and cut tail latency 40% in production.

Why kube-proxy Breaks Under Pressure

kube-proxy in iptables mode walks O(n) rules per packet. At 50k connections, each SYN traverses thousands of rules. Connection tracking locks serialize CPU cores. The result: tail latency grows linearly with connection count, not traffic volume.

IPVS mode helps but still runs in userspace. Every packet crosses the kernel/userspace boundary twice. Context switches add microseconds that compound at p99.

XDP: Drop the Packet Before the Stack Sees It

XDP (eXpress Data Path) runs eBPF programs at the NIC driver level, before skb allocation. No netfilter, no conntrack, no socket lookup. Just a deterministic hash and a redirect.

SEC("xdp")
int xdp_lb(struct xdp_md *ctx) {
    void *data = (void *)(long)ctx->data;
    void *end = (void *)(long)ctx->data_end;
    struct ethhdr *eth = data;
    if ((void*)eth + sizeof(*eth) > end) return XDP_PASS;
    if (eth->h_proto != bpf_htons(ETH_P_IP)) return XDP_PASS;
    struct iphdr *ip = data + sizeof(*eth);
    if ((void*)ip + sizeof(*ip) > end) return XDP_PASS;
    if (ip->protocol != IPPROTO_TCP) return XDP_PASS;
    struct tcphdr *tcp = (void*)ip + ip->ihl * 4;
    if ((void*)tcp + sizeof(*tcp) > end) return XDP_PASS;
    __u32 backend = bpf_htons(tcp->dest) % BACKEND_COUNT;
    return bpf_redirect_map(&backend_map, backend, 0);
}

The Backend Map

A BPF hash map stores backend MAC/IP/port tuples. Updates happen from userspace via control plane—no kernel reload. We use Cilium's LB map format for compatibility.

yaml

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: lb-pool
spec:
  cidrs:
    - 10.0.0.0/24
---
apiVersion: v1
kind: Service
metadata:
  name: api
  annotations:
    io.cilium/lb-ipam-ip: "10.0.0.100"
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080

Production Results

Metric	kube-proxy (iptables)	XDP LB	Delta
p50 latency	1.2 ms	0.9 ms	-25%
p99 latency	45 ms	27 ms	-40%
CPU (per 10k RPS)	2.1 cores	0.7 cores	-67%
Max connections	180k	1.2M	+566%

"
The first time we saw p99 flatline under load test, we thought the generator broke. It hadn't—the kernel just stopped queuing.
— Platform Lead, FinTech Unicorn

Caveats and Gotchas

⚠️

WarningXDP requires driver support (mlx5, ice, i40e, bnxt). Check `ethtool -i eth0 | grep driver`. No driver support = no XDP.

💡

TipEnable XDP_GENERIC for testing on unsupported NICs. Performance won't match native but validates logic.

Stateful services need consistent hashing (Maglev or consistent hash map). Stateless? Simple 5-tuple hash works. We use Maglev for session affinity.

Rollout Strategy

1. Deploy DaemonSet with XDP program in dry-run mode (XDP_PASS). 2. Mirror traffic to secondary interface, verify backend selection. 3. Switch DNS to XDP LB IP. 4. Drain kube-proxy. 5. Monitor `bpftool map dump` for stale entries.

ℹ️

NoteCilium 1.15+ manages this entire lifecycle. `helm install cilium --set kubeProxyReplacement=true` handles XDP attachment, map sync, and health checks.

✦

Stop tuning iptables. Start running load balancing where packets hit metal. Your p99 will thank you.

Share𝕏 Twitter in LinkedIn in Whatsapp

November 2, 2026

eBPF XDP Load Balancing Cuts Kubernetes Tail Latency 40%

Why kube-proxy Breaks Under Pressure

XDP: Drop the Packet Before the Stack Sees It

The Backend Map

Production Results

Caveats and Gotchas

Rollout Strategy

More Posts

Niantic Lightship VPS Enables Centimeter-Level Persistent AR Anchors

How WebAssembly Component Model Cuts Cold Starts 90%

Run GPUs on AWS Nitro Enclaves for Confidential AI

DNA Origami Nanostructures Deliver CRISPR Payloads Precisely

Rust Async Runtimes Compared for Low-Latency Services

Thread Border Routers Cut Matter Latency by 40 Percent

Build a Meshtastic LoRa Node for Off-Grid Texting

ROS 2 Humble DDS Tuning Cuts Planner Latency 40%

Deploying Matter 1.4 Thread Border Routers on ESP32-H2

Qiskit Runtime Primitives Cut VQE Iteration Latency 40 Percent