From: Kunwu Chan <[email protected]>
This series rounds out atomic SRCU support with a reader-free fastpath,
some atomic-context fixes, and additional test coverage.
The core change is a reader-free fastpath for synchronize_srcu_atomic().
Atomic SRCU's read-side critical sections disable preemption, so when
the summed lock and unlock counts match for both ranks,
synchronize_srcu_atomic() can end the grace period immediately, without
the index flip or the srcu_advance_state() scans. Correctness depends
on the grace-period anchor written by srcu_gp_start() preceding the
lock-count scan; two LKMM litmus tests (patches 1-2) verify that
ordering.
The series is:
1-2 LKMM litmus tests for the reader-free fastpath ordering.
3 srcutree: add the reader-free fastpath.
4 rcutorture: add --do-atomic-srcu to torture.sh.
5-7 srcutree: honor and propagate is_atomic so atomic SRCU never
transitions to big.
8 srcutree: forbid srcu_expedite_current() on atomic SRCU.
9 rcutorture: disable srcu_expedite_current() for atomic SRCU.
10 srcutree: skip callback scheduling for atomic SRCU grace periods.
11 srcutree: drop the srcu_barrier() sleep for atomic SRCU.
12 srcutree: remove leftover debug pr_alert()s.
13 srcu: restrict the non_block annotation to task context, fixing a
KCSAN data race.
The KCSAN race in patch 13 is a data race on the interrupted task's
->non_block_count when a timer callback runs atomic-SRCU readers in
inline softirq context.
Tested with herd7 7.58 (litmus tests), torture.sh --do-atomic-srcu, and
torture.sh --do-atomic-srcu --do-kcsan (KCSAN+PROVE_LOCKING).
Kunwu Chan (13):
litmus: Add SRCU fastpath anchor-before-scan test
litmus: Add SRCU fastpath scan-before-anchor test
srcutree: Add reader-free fastpath to synchronize_srcu_atomic()
rcutorture: Add atomic-SRCU support to torture.sh
srcutree: Honor is_atomic in check_init_srcu_struct()
srcutree: Make init_srcu_struct_atomic() prevent transition to big
srcutree: Don't transition atomic SRCU to big in srcu_gp_end()
srcutree: Forbid srcu_expedite_current() on atomic SRCU
rcutorture: Disable srcu_expedite_current() for atomic SRCU
srcutree: Skip callback scheduling for atomic SRCU grace periods
srcutree: Remove srcu_barrier() sleep for atomic SRCU
srcutree: Remove debug pr_alert()s
srcu: Restrict atomic-SRCU non_block annotation to task context
include/linux/srcu.h | 10 +-
kernel/rcu/rcutorture.c | 1 +
kernel/rcu/srcutree.c | 95 ++++++++++++++-----
tools/memory-model/litmus-tests/README | 16 ++++
.../SRCU-fastpath-anchor-before-scan.litmus | 56 +++++++++++
.../SRCU-fastpath-scan-before-anchor.litmus | 53 +++++++++++
.../selftests/rcutorture/bin/torture.sh | 24 +++++
7 files changed, 229 insertions(+), 26 deletions(-)
create mode 100644
tools/memory-model/litmus-tests/SRCU-fastpath-anchor-before-scan.litmus
create mode 100644
tools/memory-model/litmus-tests/SRCU-fastpath-scan-before-anchor.litmus
base-commit: ef15070f96c5f07de3a9b593c36f6e83bc9e3d58
--
2.43.0
