rcu_pending() is the gatekeeper that decides whether rcu_core() should run on the current CPU's timer tick. Currently it checks if the CPU has callbacks ready to invoke or a grace period has completed or started.
It does not check that an expedited GP has completed. After an expedited GP, callbacks remain in RCU_WAIT_TAIL (not yet advanced to RCU_DONE_TAIL) and So rcu_core() never runs to advance them. Add a check using rcu_segcblist_nextgp() combined with poll_state_synchronize_rcu_full() to detect when any pending callbacks' grace period has completed. Reviewed-by: Paul E. McKenney <[email protected]> Signed-off-by: Puranjay Mohan <[email protected]> --- kernel/rcu/tree.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0e43866dc4cd..309273a37b0a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3671,6 +3671,7 @@ EXPORT_SYMBOL_GPL(cond_synchronize_rcu_full); static int rcu_pending(int user) { bool gp_in_progress; + struct rcu_gp_oldstate gp_state; struct rcu_data *rdp = this_cpu_ptr(&rcu_data); struct rcu_node *rnp = rdp->mynode; @@ -3701,6 +3702,12 @@ static int rcu_pending(int user) rcu_segcblist_ready_cbs(&rdp->cblist)) return 1; + /* Has a GP (normal or expedited) completed for pending callbacks? */ + if (!rcu_rdp_is_offloaded(rdp) && + rcu_segcblist_nextgp(&rdp->cblist, &gp_state) && + poll_state_synchronize_rcu_full(&gp_state)) + return 1; + /* Has RCU gone idle with this CPU needing another grace period? */ if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) && !rcu_rdp_is_offloaded(rdp) && -- 2.52.0
