SRCU and Tasks RCU do not track expedited grace periods. When their callback state is checked via poll_state_synchronize_rcu_full(), the uninitialized or zeroed rgos_exp field could cause false-positive completion detection.
This commit adds an RCU_GET_STATE_NOT_TRACKED sentinel value (0x2) that these subsystems can place into rgos_exp to indicate that expedited GP tracking is not applicable. The expedited sequence check in poll_state_synchronize_rcu_full() is guarded to skip entries marked with this sentinel. This is needed to allow rcu_segcblist_advance() and rcu_accelerate_cbs() to work with both normal and expedited grace periods via get_state_synchronize_rcu_full() and poll_state_synchronize_rcu_full(). Reviewed-by: Paul E. McKenney <[email protected]> Signed-off-by: Puranjay Mohan <[email protected]> --- kernel/rcu/rcu.h | 13 +++++++++++-- kernel/rcu/tree.c | 3 ++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index fa6d30ce73d1..b450febac823 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -46,16 +46,25 @@ * the number of pending readers that will use * this inactive index is bounded). * - * RCU polled GP special control value: + * RCU polled GP special control values: * * RCU_GET_STATE_COMPLETED : State value indicating an already-completed * polled GP has completed. This value covers * both the state and the counter of the * grace-period sequence number. + * + * RCU_GET_STATE_NOT_TRACKED : State value indicating that a GP component + * is not tracked by this subsystem and should + * not be checked. Used by SRCU and RCU Tasks + * which do not track expedited GPs, to prevent + * false-positive completion when their + * gp_seq_full entries are checked via + * poll_state_synchronize_rcu_full(). */ -/* Low-order bit definition for polled grace-period APIs. */ +/* Low-order bit definitions for polled grace-period APIs. */ #define RCU_GET_STATE_COMPLETED 0x1 +#define RCU_GET_STATE_NOT_TRACKED 0x2 /* A complete grace period count */ #define RCU_SEQ_GP (RCU_SEQ_STATE_MASK + 1) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cbc170dc3f72..607fc5715cd1 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3603,7 +3603,8 @@ bool poll_state_synchronize_rcu_full(struct rcu_gp_oldstate *rgosp) if (rgosp->rgos_norm == RCU_GET_STATE_COMPLETED || rcu_seq_done_exact(&rnp->gp_seq, rgosp->rgos_norm) || rgosp->rgos_exp == RCU_GET_STATE_COMPLETED || - rcu_seq_done_exact(&rcu_state.expedited_sequence, rgosp->rgos_exp)) { + (rgosp->rgos_exp != RCU_GET_STATE_NOT_TRACKED && + rcu_seq_done_exact(&rcu_state.expedited_sequence, rgosp->rgos_exp))) { smp_mb(); /* Ensure GP ends before subsequent accesses. */ return true; } -- 2.52.0
