Jeremy reported a bug[1] while executing a BPF program containing
timed_may_goto instructions with private stacks.
timed_may_goto[2] is a runtime safety mechanism that allows BPF
programs to execute longer loops[3]. The BPF verifier replaces each
may_goto instruction with a loop counter initialized to 0xffff and a
timestamp check[4] that terminates the loop after 250 ms.
Private stacks[5] allow BPF programs to use per-CPU memory instead of
consuming more of the native kernel stack when BPF programs are deeply
nested.
To make timed may_goto work, the BPF program reserves 16 bytes of stack
space. The first 8 bytes store the loop counter and the next 8 bytes
store the timestamp.
After the loop counter is exhausted, arch_bpf_timed_may_goto() is
called. On x86, it adds the counter's stack offset to RBP to obtain a
pointer to the counter and timestamp[6]. This works when the BPF
program uses the normal stack because RBP is also the BPF frame pointer.
When a private stack is used, the x86 JIT uses R9 as the BPF frame pointer.
The verifier-generated loads and stores therefore access the counter and
timestamp through R9. However, arch_bpf_timed_may_goto() still adds the
offset to RBP and accesses an unrelated location in the native JIT stack
frame. This is the mismatch Jeremy reported.
Fix the mismatch by resolving the address in the generated BPF
instructions:
BPF_REG_AX = BPF_REG_FP
BPF_REG_AX += stack_offset
The JIT can then select the correct BPF frame pointer before calling
arch_bpf_timed_may_goto(). The function receives the resolved pointer
instead of reconstructing it from RBP.
The LoongArch timed may_goto implementation is currently queued through
the loongarch-next tree[7], while its selftests were merged separately
through the bpf-next tree[8]. This series is based on bpf-next and
therefore does not include the LoongArch trampoline update.
[1]
https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3]
https://elixir.bootlin.com/linux/v7.2.2/source/tools/testing/selftests/bpf/libarena/include/bpf_may_goto.h#L8
[4] https://elixir.bootlin.com/linux/v7.2.2/source/kernel/bpf/core.c#L3407
[5]
https://lore.kernel.org/bpf/[email protected]/
[6]
https://elixir.bootlin.com/linux/v7.2.2/source/arch/x86/net/bpf_timed_may_goto.S#L18
[7]
https://lore.kernel.org/loongarch/[email protected]/
[8] https://lore.kernel.org/bpf/[email protected]/
Siddharth Chintamaneni (7):
bpf: Fix timed may_goto stack pointer for private stacks
bpf, x86: Use resolved pointer for timed may_goto
bpf, arm64: Use resolved pointer for timed may_goto
bpf, powerpc64: Use resolved pointer for timed may_goto
bpf, riscv: Use resolved pointer for timed may_goto
bpf, s390: Use resolved pointer for timed may_goto
selftests/bpf: Test timed may_goto with private stacks
arch/arm64/net/bpf_timed_may_goto.S | 12 ++------
arch/powerpc/net/bpf_timed_may_goto.S | 8 ++---
arch/riscv/net/bpf_timed_may_goto.S | 13 ++++----
arch/s390/net/bpf_jit_comp.c | 6 ++--
arch/s390/net/bpf_timed_may_goto.S | 8 ++---
arch/x86/net/bpf_timed_may_goto.S | 6 ----
kernel/bpf/fixups.c | 19 ++++++------
.../bpf/progs/verifier_bpf_fastcall.c | 30 ++++++++++---------
.../selftests/bpf/progs/verifier_may_goto_1.c | 17 ++++++-----
.../bpf/progs/verifier_private_stack.c | 19 ++++++++++++
10 files changed, 75 insertions(+), 63 deletions(-)
base-commit: d761934c9483ecde93fe99d8705282f716dfee50
--
2.43.0
