This series extends Paul McKenney's v3 hazptr implementation [1] with a shared scan path for concurrent hazptr_synchronize() callers, and adapts the lockdep dynamic-key hashlist use case from Boqun Feng's 2025 shazptr series [2] to the current hazptr API.
The series also adds rcuscale support and torture coverage for the hazptr implementation. The lockdep conversion replaces the expedited RCU wait in lockdep_unregister_key() with hazptr_synchronize(). This limits the wait to hazard pointers protecting the target hash bucket instead of waiting for a system-wide expedited RCU grace period. [1] https://lore.kernel.org/all/[email protected]/ [2] https://lore.kernel.org/lkml/[email protected]/ Performance data ================ All measurements are on an ARM64 KVM guest with HAZPTR_SCALE_NR_OBJS=8 and round-robin updater selection; unless noted otherwise, nreaders=0. Scale tests run with PROVE_LOCKING=n. rcuscale synchronize latency (96 CPUs, nw=1): nreaders=0 nreaders=1 nreaders=4 nreaders=96 scale avg avg avg avg -------------------------------------------------------------------- hazptr 46 us 47 us 102 us 106 ms RCU 8.3 ms 8.5 ms 16.6 ms 54.4 ms SRCU 8.2 ms 8.0 ms 11.7 ms 11.8 ms Hazptr single-writer latency remains below 110 us with up to four readers on this 96-CPU guest, but rises to 106 ms when all 96 CPUs hold hazard pointers. Under 16 concurrent synchronize callers (nw=16), per-writer latency at four CPU counts: hazptr nw=16 RCU nw=16 SRCU nw=16 CPUs avg avg avg ----------------------------------------------------------- 24 8.0 ms 13.6 ms 9.2 ms 96 8.0 ms 13.3 ms 8.6 ms 128 7.9 ms 13.0 ms 11.7 ms 256 8.0 ms 20.8 ms 9.6 ms Hazptr remains around 8.0 ms across these CPU counts, with the scan-kthread retry interval contributing to the latency. RCU rises to ~21 ms at 256 CPUs while SRCU stays around 9-12 ms. Reader-side overhead (refscale, 96 readers on the 96-CPU guest, 3 runs each): hazptr 25.7 ns/op RCU 96.7 ns/op SRCU 134.6 ns/op lockdep workload -- tc qdisc mq x100, ARM64 KVM, PROVE_LOCKING=y, 96 background hazptr readers, function-call IPIs per 100 ops: CPUs hazptr exp RCU reduction ------------------------------------------------------------ 24 196 207 5% 96 189 265 29% 128 193 266 27% 256 222 384 42% Hazptr issues fewer function-call IPIs at each CPU count, with the difference reaching 42% at 256 CPUs. The hazptr.sh test suite passes, including the lockdep scenarios and the 8-to-256 CPU sweep, with no lockdep warnings, deadlocks, or crashes. Additional x86 server testing with Lian Wang is planned(maybe after LPC). Changes since RFC/WIP ===================== - RFC/WIP: https://lore.kernel.org/all/[email protected]/ - Split the original 4-patch RFC/WIP into smaller commits covering shared scanning, correctness, API support, lockdep, scaling, and torture testing. - Incorporated Boqun Feng's review feedback: use a Bloom filter to avoid per-waiter allocation, add scoped_guard() support, and add a debug option to force the hazptr acquire slow path. - Fixed scan ordering around backup-slot promotion by scanning all per-CPU slots before the overflow lists, with a separate overflow-list phase. - Simplified the scan cycle to flip first and drain only the old wildcard generation, with herd7-verified LKMM tests for both the in-flight and resolved publication cases. - Extended rcuscale and hazptrtorture coverage, added a selftest script for the torture configurations, and fixed the hazptr_release() kernel-doc. Kunwu Chan (15): hazptr: add shared scan kthread hazptr: use Bloom filter for shared scan waiters hazptr: scan all per-CPU slots before overflow lists hazptr: add scoped_guard() support hazptr: add debug option to force the acquire slow path hazptr: elide redundant first drain pass Documentation/litmus-tests: add hazptr wildcard-flip escape test locking/lockdep: use hazptr to wait for dynamic key lookups rcuscale: add hazptr scale type hazptr: fix kernel-doc of hazptr_release() Documentation/litmus-tests: add hazptr acquire-before-scan test hazptrtorture: add slowpath and lockdep scenarios hazptrtorture: add READERS4 and READERS0 torture configs hazptrtorture: add 128- and 256-CPU configs selftests/rcutorture: add hazptr torture test script Documentation/litmus-tests/README | 13 + .../hazptr/hazptr-acquire-before-scan.litmus | 45 +++ .../hazptr/hazptr-wildcard-flip-escape.litmus | 46 +++ include/linux/hazptr.h | 56 ++- kernel/hazptr.c | 336 +++++++++++++++++- kernel/locking/lockdep.c | 25 +- kernel/rcu/Kconfig.debug | 10 + kernel/rcu/hazptrtorture.c | 57 ++- kernel/rcu/rcuscale.c | 70 +++- .../selftests/rcutorture/bin/hazptr.sh | 146 ++++++++ .../rcutorture/configs/hazptr/CFLIST | 6 + .../rcutorture/configs/hazptr/CPU128 | 16 + .../rcutorture/configs/hazptr/CPU128.boot | 1 + .../rcutorture/configs/hazptr/CPU256 | 16 + .../rcutorture/configs/hazptr/CPU256.boot | 1 + .../rcutorture/configs/hazptr/LOCKDEP | 17 + .../rcutorture/configs/hazptr/LOCKDEP.boot | 1 + .../rcutorture/configs/hazptr/READERS0 | 16 + .../rcutorture/configs/hazptr/READERS0.boot | 2 + .../rcutorture/configs/hazptr/READERS4 | 16 + .../rcutorture/configs/hazptr/READERS4.boot | 2 + .../rcutorture/configs/hazptr/SLOWPATH | 16 + 22 files changed, 881 insertions(+), 33 deletions(-) create mode 100644 Documentation/litmus-tests/hazptr/hazptr-acquire-before-scan.litmus create mode 100644 Documentation/litmus-tests/hazptr/hazptr-wildcard-flip-escape.litmus create mode 100755 tools/testing/selftests/rcutorture/bin/hazptr.sh create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU128 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU128.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU256 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/CPU256.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/LOCKDEP create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/LOCKDEP.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS0 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS0.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS4 create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/READERS4.boot create mode 100644 tools/testing/selftests/rcutorture/configs/hazptr/SLOWPATH -- 2.43.0
