Brussels / 1 & 2 February 2020


Tools and mechanisms to debug BPF programs

By allowing to safely load programs from user space and to execute them in the kernel, eBPF (extended Berkeley Packet Filter) has brought new possibilities to the Linux kernel, in particular in terms of tracing and network processing.

But when a program fails to load, or when it does not return the expected values, what tools do we have to examine, inspect and debug eBPF objects? This talk focuses on the different tools and mechanisms available to help eBPF developers debug their programs, at the different stages of the workflow. From bpftool to test runs, let's find the best way to track bugs!

I am not sure how long the time slots for the debugging devroom are, so the content would be adapted according to the duration of the talk. The idea is to:

  1. Very briefly introduce the existing tools/mechanisms for debugging BPF programs at each stage of the workflow
  2. Spend some time on a more in-depth introduction to bpftool, which can be used to perform a variety of operations on eBPF programs, maps, or BTF objects
  3. Mention leads for future work

In more details, this would look like:

  1. Different tools to debug

1.1 Compilation time (make sure program is generated as intended)

  • LLVM backend
  • llvm-objdump
  • eBPF assembly

1.2 Load time / Verifier / JIT compile time (make sure program loads successfully)

  • libbpf / ip / tc
  • verifier logs, kernel logs, extack messages
  • Documentation
  • bpftool

1.3 Runtime (make sure program return as expected)

  • bpftool
  • bpftraceprintk() helper / perf events
  • (user space BPF machines)
  • bpftool prog run
  • perf annotations
  • BTF debug information
  • BPF trampolines: spy on BPF with BPF

1.4 User space (develop programs that manipulate BPF objects)

  • strace, valgrind support for bpf()
  • probing kernel with bpftool
  • using tools to generate BPF (bcc, bpftrace, libkefir, ...)

  • bpftool introduction (with examples)

  • listing objects

  • loading programs, attaching programs, creating maps
  • performing “test runs”
  • probing kernel for existing features
  • taking over the world

  • In the future?

  • checking a program loads: bpftool prog probe object_file.o

  • debugger: break points, dump for registers/stack/context? (Complete the support (hooks exist) in GDB? Extend BPFPROGTEST_RUN infrastructure?)


Quentin Monnet