[PATCH tip/core/rcu 08/40] rcu: Make call_rcu() do deferred NOCB wakeups

[Paul E. McKenney]

If a CONFIG_RCU_NOCB_CPUS kernel invokes call_rcu() with interrupts
disabled, wakeups must be deferred in order to avoid self-deadlock in the
cases where the disabled interrupts are due to scheduler locks being held.
In this case, a flag is set and is checked on entry to extended quiescent
states (usermode, idle), on exit from the RCU_SOFTIRQ handler, and when
the CPU in question goes offline.  However, a given CPU could avoid all
of those states for a considerable length of time.

This commit therefore allows a subsequent invocation of call_rcu() with
interrupts enabled to do the wakeup.  It also makes the wakeup function
clear the deferred-wakeup flag.

Signed-off-by: Paul E. McKenney 
---
 kernel/rcu/tree_plugin.h | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 0a62a8f1caac..5a149091d262 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1768,6 +1768,7 @@ static void wake_nocb_leader(struct rcu_data *rdp, bool 
force)
return;
if (READ_ONCE(rdp_leader->nocb_leader_sleep) || force) {
/* Prior smp_mb__after_atomic() orders against prior enqueue. */
+   WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE_NOT);
WRITE_ONCE(rdp_leader->nocb_leader_sleep, false);
swake_up(_leader->nocb_wq);
}
@@ -1853,7 +1854,9 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp,
return;
}
len = atomic_long_read(>nocb_q_count);
-   if (old_rhpp == >nocb_head) {
+   if (old_rhpp == >nocb_head ||
+   (rcu_nocb_need_deferred_wakeup(rdp) &&
+!irqs_disabled_flags(flags))) {
if (!irqs_disabled_flags(flags)) {
/* ... if queue was empty ... */
wake_nocb_leader(rdp, false);
-- 
2.5.2

[PATCH tip/core/rcu 08/40] rcu: Make call_rcu() do deferred NOCB wakeups

[Paul E. McKenney]

If a CONFIG_RCU_NOCB_CPUS kernel invokes call_rcu() with interrupts
disabled, wakeups must be deferred in order to avoid self-deadlock in the
cases where the disabled interrupts are due to scheduler locks being held.
In this case, a flag is set and is checked on entry to extended quiescent
states (usermode, idle), on exit from the RCU_SOFTIRQ handler, and when
the CPU in question goes offline.  However, a given CPU could avoid all
of those states for a considerable length of time.

This commit therefore allows a subsequent invocation of call_rcu() with
interrupts enabled to do the wakeup.  It also makes the wakeup function
clear the deferred-wakeup flag.

Signed-off-by: Paul E. McKenney 
---
 kernel/rcu/tree_plugin.h | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 0a62a8f1caac..5a149091d262 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1768,6 +1768,7 @@ static void wake_nocb_leader(struct rcu_data *rdp, bool 
force)
return;
if (READ_ONCE(rdp_leader->nocb_leader_sleep) || force) {
/* Prior smp_mb__after_atomic() orders against prior enqueue. */
+   WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE_NOT);
WRITE_ONCE(rdp_leader->nocb_leader_sleep, false);
swake_up(_leader->nocb_wq);
}
@@ -1853,7 +1854,9 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp,
return;
}
len = atomic_long_read(>nocb_q_count);
-   if (old_rhpp == >nocb_head) {
+   if (old_rhpp == >nocb_head ||
+   (rcu_nocb_need_deferred_wakeup(rdp) &&
+!irqs_disabled_flags(flags))) {
if (!irqs_disabled_flags(flags)) {
/* ... if queue was empty ... */
wake_nocb_leader(rdp, false);
-- 
2.5.2