The old grace-period start code would acquire only the leaf's rcu_node structure's ->lock if that structure believed that a grace period was in progress. The new code advances to the leaf's parent in this case, needlessly acquiring then leaf's parent's ->lock. This commit therefore checks the grace-period state after marking the leaf with the need for the specified grace period, and if the leaf believes that a grace period is in progress, takes an early exit.
Reported-by: Joel Fernandes <[email protected]> Signed-off-by: Paul E. McKenney <[email protected]> [ paulmck: Add "Startedleaf" tracing as suggested by Joel Fernandes. ] --- include/trace/events/rcu.h | 4 ++-- kernel/rcu/tree.c | 9 +++++++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 7d3650cc9d30..5ab1df0a2801 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -91,8 +91,8 @@ TRACE_EVENT(rcu_grace_period, * * "Startleaf": Request a grace period based on leaf-node data. * "Prestarted": Someone beat us to the request - * "Startedleaf": Leaf-node start proved sufficient. - * "Startedleafroot": Leaf-node start proved sufficient after checking root. + * "Startedleaf": Leaf node marked for future GP. + * "Startedleafroot": All nodes from leaf to root marked for future GP. * "Startedroot": Requested a nocb grace period based on root-node data. * "NoGPkthread": The RCU grace-period kthread has not yet started. * "StartWait": Start waiting for the requested grace period. diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index fe986771c5cf..1c2aa020688a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1590,6 +1590,15 @@ static bool rcu_start_this_gp(struct rcu_node *rnp, struct rcu_data *rdp, goto unlock_out; } rnp_root->gp_seq_needed = c; + if (rcu_seq_state(rcu_seq_current(&rnp->gp_seq))) { + /* + * We just marked the leaf, and a grace period + * is in progress, which means that rcu_gp_cleanup() + * will see the marking. Bail to reduce contention. + */ + trace_rcu_this_gp(rnp, rdp, c, TPS("Startedleaf")); + goto unlock_out; + } if (rnp_root != rnp && rnp_root->parent != NULL) raw_spin_unlock_rcu_node(rnp_root); if (!rnp_root->parent) -- 2.17.1
