On Fri Sep 18, 2026 at 2:49 PM UTC, Josef Bacik wrote: > On HAVE_RCU_TRAMPOLINE_READERS kernels Tasks RCU keeps a BPF trampoline > image allocated only while a task using it is a Tasks Trace RCU reader > or is executing text that rcu_tasks_trampoline_text() recognises. The > image itself is such text, but the C glue and the programs it calls are > not, and only sleepable programs take rcu_read_lock_trace() today. > > Have the x86-64 JIT open-code rcu_read_lock_trace() and > rcu_read_unlock_trace() in the trampoline, as ftrace_64.S does for > ftrace_caller: one reader from just after the frame is set up to just > before the original function is called, covering __bpf_tramp_enter() > and the fentry and fmod_ret programs, and a second one from just after > the original function returns to just before the final register > restore, covering the fexit programs and __bpf_tramp_exit(). The > original function itself runs outside both, since it may run for a long > time and the image is pinned by im->pcref across it. Trampolines that > do not call the original function get a single reader around all their > programs. The second reader is entered before ip_after_call, so the > ip_after_call -> ip_epilogue jump that bpf_tramp_image_put() patches in > is inside it, and the fmod_ret early-exit branch lands after that point > still holding the first reader, so exactly one is held on every path. > > The sequence uses r10 and r11, which are scratch at each emission point, > and references current_task and rcu_tasks_trace_srcu_struct by absolute > sign-extended address, the form the JIT already relies on for > this_cpu_off. Sleepable programs' own rcu_read_lock_trace() simply > nests. Nothing is emitted on other configurations. > > Suggested-by: Alexei Starovoitov <[email protected]> > Assisted-by: LLM > Signed-off-by: Josef Bacik <[email protected]> > --- > arch/x86/net/bpf_jit_comp.c | 113 > ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 113 insertions(+) > > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c > index 2853e87797a7..c991f7ceacdf 100644 > --- a/arch/x86/net/bpf_jit_comp.c > +++ b/arch/x86/net/bpf_jit_comp.c > @@ -14,6 +14,7 @@ > #include <linux/memory.h> > #include <linux/sort.h> > #include <linux/execmem.h> > +#include <linux/rcupdate_trace.h> > #include <asm/extable.h> > #include <asm/ftrace.h> > #include <asm/set_memory.h> > @@ -722,6 +723,97 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg, > u8 *ip) > *pprog = prog; > } > > +/* > + * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the > + * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent > + * macros in arch/x86/kernel/ftrace_64.S. The image is not relocated, so > + * current_task and rcu_tasks_trace_srcu_struct are referenced by absolute > + * (sign-extended 32-bit) address, the form the JIT already relies on for > + * this_cpu_off. Uses r10 and r11, which are scratch at every emission > + * point, and clobbers flags. > + * > + * lock: unlock: > + * mov r11, gs:[current_task] mov r11, gs:[current_task] > + * mov r10d, [r11+nesting] mov r10d, [r11+nesting] > + * inc dword ptr [r11+nesting] sub r10d, 1 > + * test r10d, r10d jnz 2f > + * jnz 1f mov r10, [r11+scp] > + * mov r10, [&srcu.srcu_ctrp] mov dword ptr [r11+nesting], 0 > + * inc qword ptr gs:[r10+locks] (smp_mb) > + * mov [r11+scp], r10 inc qword ptr gs:[r10+unlocks] > + * (smp_mb) jmp 3f > + * 1: 2: mov [r11+nesting], r10d > + * 3: > + */ > +static void emit_trace_rcu_reader(u8 **pprog, bool lock) > +{ > +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS > + const u32 nesting = offsetof(struct task_struct, trc_reader_nesting); > + const u32 scp = offsetof(struct task_struct, trc_reader_scp);
nesting_off ? scp_off ? Otherwise EMIT(nesting, 4); is a bit confusing. > + const bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB); > + u8 *prog = *pprog; > + > + BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)); > + BUILD_BUG_ON(offsetof(struct srcu_ctr, srcu_locks) != 0); > + BUILD_BUG_ON(offsetof(struct srcu_ctr, srcu_unlocks) != 8); Looks too hardcoded here. Why not to use the same approach as with offsetof() few lines above? Overall looks ok, but I wonder what Paul will say that rcu_read_lock_tasks_trace() becomes baked in into JITs and will be pretty hard to change. We also lose rcu_tt lockdep runtime checks. So lockdep might get confused?
