On 9/7/2026 11:46 AM, Masami Hiramatsu (Google) wrote: > Hi, > > Here is the 15th version of the series for adding new wprobe (watch probe) > which provides memory access tracing event. Moreover, this can be used > via event trigger. Thus it can trace memory access on dynamically > allocated objects too. > The previous version is here: > > > https://lore.kernel.org/all/178810001186.64882.2161016469449127450.stgit@devnote2/ > > In this version, previous bugfix patches for BTF kflag resolution and > RCU-protected kprobe blacklist were sent to Linus via probe/fixes, reducing > the series to 12 patches. This version addresses feedbacks from Sashiko on > v14[1], and more local fixes such as x86 debug register manipulation, > validates > watched address and natural alignment, tracks missed events on SMP update > failures(*), prevents in-place command string modification during trigger > parsing, adds explicit failure checks in selftests, and documents BTF > typecast and struct offset syntax. I also added Assited-by: tags for this > update. > > [1] > https://sashiko.dev/#/patchset/178810001186.64882.2161016469449127450.stgit%40devnote2 > (*) currently it just increase a missed counter. > > Detailed changes are here. > > Changes in v15: > - x86/mce: Move local_db_save() to entry of exc_machine_check_kernel() > and restore it on early return from mce_check_crashing_cpu() to > prevent nested #DB on watched variables. > - x86/hw_breakpoints: Always preserve DR7_FIXED_1 in > arch_install_hw_breakpoint(), arch_uninstall_hw_breakpoint(), and > hw_breakpoint_restore() for architectural compliance. > - x86/hw_breakpoints: Wrap local_db_save() in a cpu_dr7_seq loop to > detect NMI breakpoint updates and ensure DR7 is disabled before entry. > - x86/hw_breakpoints: In local_db_restore(), restore DR7_FIXED_1 if > saved dr7 is 0 and an NMI updated breakpoints, preserving KVM guest > isolation. > - x86/hw_breakpoints: Always write "val | DR7_FIXED_1" to hardware DR7 > in local_db_restore() for architectural compliance. > - x86/hw_breakpoints: Avoid modifying saved dr7 in-place across retry > loops. > - x86/hw_breakpoints: Temporarily clear breakpoint enable bit in DR7 > before updating debug address register and re-enable it afterward to > comply with x86 specification and avoid spurious #DBs. > - HWBP: Annotate counter_arch_bp(bp)->address and bp->attr.bp_addr with > WRITE_ONCE() in both update and rollback paths to prevent data races > with concurrent readers (e.g. NMI handlers). > - tracing/wprobe: Introduce trace_wprobe_is_valid_addr() to validate > target address and natural alignment for wprobe length during event > creation and trigger setup. > - tracing/wprobe: Initialize irq_work and work immediately after > allocation in alloc_trace_wprobe() and check tw->work.func in > free_trace_wprobe() to prevent WARN_ON(!work->func) on early failures. > - tracing/wprobe: Record SMP local update failures in tw->missed counter > and expose missed event count as a trailing comment in trigger file. > - tracing/wprobe: Duplicate field string in wprobe_trigger_typecast_parse() > to avoid modifying glob in place so tracing_log_err() logs the pristine > command string. > - tracing/wprobe: Remove leftover offset and adjust debug print in > wprobe_trigger_print(). > - tracing/wprobe: Add set_wprobe and clear_wprobe triggers to readme_msg > in trace.c. > - tracing/wprobe: Move variable declarations to function entry in > trace_wprobe_show(), fix tab indentation in trace.c, and wrap lines > exceeding 100 columns. > - Documentation: Document BTF typecast in fetchargs and BTF struct > offset resolution in set_wprobe trigger in wprobetrace.rst. > - selftests/ftrace: Add explicit failure checks with fail helper across > trigger-wprobe.tc, trigger-wprobe-btf-typecast.tc, and > trigger-wprobe-btf-offset.tc. > - selftests/ftrace: Declare fprobe README requirement in trigger tests > and trigger file check in trigger-wprobe-syntax-errors.tc requires line. > > > Public branch > ------------- > I will push this branch as topic/wprobe-v2 to my tree so that it > can be easily tested. > > https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/wprobe-v2 > > This is based on linux-trace tree's linux-trace/probes/fixes. > > Usage > ----- > > The basic usage of this wprobe is similar to other probes; > > w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+|-OFFS]>[:LEN] [FETCHARGS] > > This defines a new wprobe event. For example, to trace jiffies update, > you can do; > > echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events > echo 1 > events/wprobes/my_jiffies/enable > > Moreover, this can be combined with event trigger to trace the memory > access on slab objects. The trigger syntax is; > > set_wprobe:WPROBE_EVENT:FIELD[+|-OFFSET][:COUNT] [if FILTER] > clear_wprobe:WPROBE_EVENT[:FIELD[+|-OFFSET][:COUNT]] [if FILTER] > > set_wprobe sets WPROBE_EVENT's watch address on FIELD[+|-OFFSET]. > clear_wprobe clears WPROBE_EVENT's watch address if it is set to > FIELD[+|-OFFSET]. If FIELD is omitted, forcibly clear the watch address > when trigger event is hit. > > For example, trace the first 8 byte of the dentry data structure passed > to do_truncate() until it is deleted by dentry_kill(). > (Note: all tracefs setup uses '>>' so that it does not kick do_truncate()) > > # echo 'w:watch rw@-1:8 address=$addr value=+0($addr)' > dynamic_events > > # echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events > # echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger > > # echo 'f:dentry_kill dentry_kill dentry=$arg1' >> dynamic_events > # echo 'clear_wprobe:watch:dentry' >> events/fprobes/dentry_kill/trigger > > # echo 1 >> events/fprobes/truncate/enable > # echo 1 >> events/fprobes/dentry_kill/enable > > # echo aaa > /tmp/hoge > # echo bbb > /tmp/hoge > # echo ccc > /tmp/hoge > # rm /tmp/hoge > > Then, the trace data will show; > > # tracer: nop > # > # entries-in-buffer/entries-written: 32/32 #P:8 > # > # _-----=> irqs-off/BH-disabled > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / _-=> migrate-disable > # |||| / delay > # TASK-PID CPU# ||||| TIMESTAMP FUNCTION > # | | | ||||| | | > sh-107 [004] ...1. 9.990418: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618 > sh-107 [004] ...1. 9.990914: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004b3de78 > sh-107 [004] ...1. 9.993175: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880049ddd40 > sh-107 [004] ..... 9.995198: truncate: > (do_truncate+0x4/0x120) dentry=0xffff8880048083a8 > sh-107 [004] ...1. 9.995389: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db998 > sh-107 [004] ..Zff 9.997503: watch: > (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080 > sh-107 [004] ..Zff 9.997509: watch: > (path_openat+0x211/0xda0) address=0xffff8880048083a8 value=0x8200080 > sh-107 [004] ..Zff 9.997514: watch: > (path_openat+0xa56/0xda0) address=0xffff8880048083a8 value=0x8200080 > sh-107 [004] ..Zff 9.997518: watch: > (path_openat+0xae2/0xda0) address=0xffff8880048083a8 value=0x8200080 > sh-107 [004] ..... 9.997521: truncate: > (do_truncate+0x4/0x120) dentry=0xffff8880048083a8 > sh-107 [004] ...1. 9.997582: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004808270 > sh-107 [004] ...1. 9.999365: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db728 > sh-107 [004] ...1. 9.999388: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004b1c000 > rm-113 [005] ..Zff 10.000965: watch: > (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.000971: watch: > (path_lookupat+0x97/0x1e0) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.000984: watch: > (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.000988: watch: > (path_lookupat+0x97/0x1e0) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.001010: watch: > (lookup_one_qstr_excl+0x28/0x140) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.001014: watch: > (lookup_one_qstr_excl+0xd1/0x140) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.001018: watch: > (may_delete_dentry+0x1c/0x200) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.001021: watch: > (may_delete_dentry+0x195/0x200) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] ..Zff 10.001031: watch: > (vfs_unlink+0x5e/0x260) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] d.Z.. 10.001067: watch: > (d_make_discardable+0x1b/0x40) address=0xffff8880048083a8 value=0x8200080 > rm-113 [005] d.Z.. 10.001071: watch: > (d_make_discardable+0x29/0x40) address=0xffff8880048083a8 value=0x200080 > rm-113 [005] ...1. 10.001072: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880048083a8 > rm-113 [005] ...1. 10.001218: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880048083a8 > sh-107 [004] ...1. 10.001416: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db110 > sh-107 [004] ...1. 10.001444: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db248 > sh-107 [004] ...1. 10.001500: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618 > sh-107 [004] ...1. 10.002067: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004b41e78 > sh-107 [004] ...1. 10.904920: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004b41e78 > sh-107 [004] ...1. 10.905129: dentry_kill: > (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618 > > > Thank you, > > --- > base-commit: 0c4256196b3a105307e2235fbfd85e768bbcdd0f > > Jinchao Wang (2): > x86/hw_breakpoints: Make DR7 updates NMI safe > x86/hw_breakpoints: Add arch_modify_local_hw_breakpoint_addr() API > > Masami Hiramatsu (Google) (10): > x86/mce: Fix hardware debug register corruption on task migration > HWBP: Add modify_local_hw_breakpoint_addr() API > tracing/wprobe: Add wprobe (watchpoint probe) trace event support > x86: hw_breakpoint: Add a kconfig to clarify when a breakpoint fires > selftests: tracing: Add a basic testcase for wprobe > selftests: tracing: Add syntax testcase for wprobe > tracing/wprobe: Add set_wprobe and clear_wprobe event triggers > selftests: tracing: Add wprobe trigger testcase > tracing/wprobe: Support BTF typecast in fetchargs > tracing/wprobe: Support BTF struct offset resolution in set_wprobe > trigger > > > Documentation/trace/index.rst | 1 > Documentation/trace/wprobetrace.rst | 200 +++ > arch/Kconfig | 18 > arch/x86/Kconfig | 2 > arch/x86/include/asm/debugreg.h | 67 + > arch/x86/include/asm/hw_breakpoint.h | 2 > arch/x86/kernel/cpu/mce/core.c | 29 > arch/x86/kernel/hw_breakpoint.c | 151 +- > arch/x86/kernel/nmi.c | 7 > arch/x86/kernel/traps.c | 10 > include/linux/hw_breakpoint.h | 6 > include/linux/trace_events.h | 3 > kernel/events/hw_breakpoint.c | 48 + > kernel/trace/Kconfig | 24 > kernel/trace/Makefile | 1 > kernel/trace/trace.c | 13 > kernel/trace/trace.h | 6 > kernel/trace/trace_events_trigger.c | 2 > kernel/trace/trace_probe.c | 36 > kernel/trace/trace_probe.h | 22 > kernel/trace/trace_wprobe.c | 1522 > ++++++++++++++++++++ > tools/testing/selftests/ftrace/config | 3 > .../ftrace/test.d/dynevent/add_remove_wprobe.tc | 63 + > .../test.d/dynevent/wprobes_syntax_errors.tc | 23 > .../test.d/trigger/trigger-wprobe-btf-offset.tc | 85 + > .../test.d/trigger/trigger-wprobe-btf-typecast.tc | 85 + > .../test.d/trigger/trigger-wprobe-syntax-errors.tc | 37 > .../ftrace/test.d/trigger/trigger-wprobe.tc | 107 + > 28 files changed, 2476 insertions(+), 97 deletions(-) > create mode 100644 Documentation/trace/wprobetrace.rst > create mode 100644 kernel/trace/trace_wprobe.c > create mode 100644 > tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc > create mode 100644 > tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc > create mode 100644 > tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-offset.tc > create mode 100644 > tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-typecast.tc > create mode 100644 > tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-syntax-errors.tc > create mode 100644 > tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe.tc > > -- > Masami Hiramatsu (Google) <[email protected]> Hi Masami,
I see the watch address update currently goes irq_work -> workqueue -> on_each_cpu(). I would like to discuss that path. The race window wprobe may be used to watch can be very narrow -- sometimes it lives entirely inside a single function -- and the trigger does not always run in in_task() context. The current path defers the update to a work item, so by the time the debug registers are updated the window may already be gone. Now that we have modify_local_hw_breakpoint_addr(), the trigger could arm the local CPU directly and use smp_call_function_single_async() for the remote ones. That should improve the overlap between the watch and the window. What do you think? Thanks, Jinchao
