How Do I Profile eBPF Code?

Jul 28, 2026 10:55 PM - 2 hours ago 76

If we are moving immoderate eBPF workload aliases penning eBPF code, we want to measurement its capacity impact, and successful this station we will show an illustration of really to do it.

In this example, our extremity is to measurement the capacity of record unfastened operations, 1 of the astir captious functions successful the OS. Our codification utilized record unfastened hooks successful eBPF, we wanted to measurement the capacity overhead introduced by adding this hook.

To place apt bottlenecks, we request a elemental C trial harness pinch the fewest dependencies, designed to measurement record unfastened performance.

#define _GNU_SOURCE #include <fcntl.h> #include <stdio.h> #include <time.h> #include <stdint.h> #include <stdlib.h> #include <unistd.h> #include <sched.h> #include <sys/mman.h> #include <sys/syscall.h> static inline uint64_t now_ns(void) { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); /* VDSO, nary syscall */ return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec; } int main(int argc, char **argv) { const char *path = argv[1]; uint64_t n = strtoull(argv[2], NULL, 10); uint64_t lukewarm = n / 10; /* preallocate + prefault + lock: nary page faults successful the loop */ uint32_t *d = mmap(NULL, n * sizeof(uint32_t), PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); mlock(d, n * sizeof(uint32_t)); for (uint64_t one = 0; one < n; i++) d[i] = 0; /* responsibility everything successful */ for (uint64_t one = 0; one < n; i++) { uint64_t t0 = now_ns(); long fd = syscall(SYS_openat, AT_FDCWD, path, O_RDONLY); /* raw, nary libc wrapper */ uint64_t t1 = now_ns(); if (fd >= 0) close(fd); d[i] = (uint32_t)(t1 - t0); } /* dump aft the loop only */ for (uint64_t one = warm; one < n; i++) printf("%u\n", d[i]); return 0; }

The extremity of the codification illustration supra is to support it elemental and reopen the aforesaid record nether lukewarm cache conditions, minimizing unrelated filesystem and disk I/O variability and helping place the p50/p99. The codification invokes syscall(SYS_openat, …) alternatively of the libc openat() wrapper, and the first 10% of the results are discarded arsenic a warmup period.

This trial harness would nutrient results of opening a record x number of times and really agelong it took to open. So we could usage this to measurement the earlier and aft of erstwhile the eBPF hook attached.

Setup

When profiling the eBPF code, we want perf instrumentality to beryllium capable to resoluteness symbols truthful we tin analyse wherever the rumor is successful our code. To do that, we person to tally these commands.

sudo sysctl -w net.core.bpf_jit_enable=1 sudo sysctl -w net.core.bpf_jit_kallsyms=1

The supra commands alteration jit and expose jit-compiled BPF symbols truthful the perf study tin show programme names alternatively of chartless addresses.

To cheque whether the symbols look successful the perf tool, tally your eBPF codification and usage a bid for illustration this.

In the supra rg command, we are checking for lsm arsenic we are measuring LSM hooks.

Also, we are utilizing a civilization kernel version, truthful perf for that kernel type is not successful the modular path, and we person it installed successful our example: PERF=/usr/lib/linux-tools/6.8.0-134-generic/perf

Measuring

Now that we person group up each the basal tools, the first measurement is to measurement without the eBPF codification running, and this is wherever utilizing the supra C codification tin help. We measurement the codification by opening the record /etc/hostname and piping the results to a record truthful that we tin cipher the p50/p99.

sudo taskset -c 3 chrt -f 99 ./bench /etc/hostname 100000 > /tmp/samples.txt

The taskset -c 3 pins execution to CPU 3, reducing CPU migration noise, and chrt -f 99 gives the benchmark highly precocious CPU priority. It runs earlier almost each normal programs and keeps moving until it finishes, blocks, aliases is interrupted. The C codification discards the first 10%; the record should incorporate 90,000 samples.txt.

Next, tally the eBPF codification and execute thing for illustration this.

sudo $PERF grounds \ -g \ --call-graph fp \ -e cycles:k \ -F 997 \ -o ~/perf.data \ -- \ taskset -c 3 \ chrt -f 99 \ ./bench /etc/hostname 200000 \ > /tmp/samples.txt

The -g records the telephone stacks, --call-graph fp unwinds stacks utilizing framework pointers, and -e samples CPU cycles successful kernel mode only, which includes syscall, VFS, LSM, and eBPF execution and not userspace benchmark work. The -F 997 requests 997 samples per second, and the non-round wave helps debar periodic alignment.

After moving the above, tally this bid to benignant the data.

sudo "$PERF" study -i ~/perf.data --stdio --sort comm,dso,symbol > perf.txt

alt text

Flamegraph of the aforesaid perf.data, generated pinch Inferno. The stack of liking present is bpf_lsm_file_open and everything supra it.

Here is an illustration output from perf.txt, which shows that the clip being spent connected bpf_lsm_file_open and its tail calls is wherever the capacity bottleneck is. This turned retired to beryllium successful a basking path, which meant each allocation-to-CPU rhythm shaving will make a important effect connected the capacity of the system.

This station focuses connected the profiling method alternatively than a circumstantial result, and the overhead you’ll spot depends heavy connected what your hook really does, truthful we’re leaving the numbers retired and focusing connected really to get them yourself.

Now, from the above, we tin commencement analyzing wherever the clip is being spent and perf-tune the codification on pinch the p50/p99 of the C codification pinch eBPF running.

By doing this, we tin intelligibly place the perf effect of the eBPF codification and apt pinpoint wherever optimizations are required. It tin beryllium arsenic elemental arsenic caching thing aliases coming up pinch a amended algorithm based connected wherever the problem is.