When we're in hard interrupt context in rcu_read_unlock_special(), we can still benefit from invoke_rcu_core() doing wake ups of rcuc threads when the !use_softirq parameter is passed. This is safe to do so because:
1. We avoid the scheduler deadlock issues thanks to the deferred_qs bit introduced in commit 23634ebc1d94 ("rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()") by checking for the same in this patch. 2. in_irq() implies in_interrupt() which implies raising softirq will not do any wake ups. The rcuc thread which is awakened will run when the interrupt returns. We also honor 25102de ("rcu: Only do rcu_read_unlock_special() wakeups if expedited") thus doing the rcuc awakening only when none of the following are true: 1. Critical section is blocking an expedited GP. 2. A nohz_full CPU. If neither of these cases are true (exp == false), then the "else" block will run to do the irq_work stuff. This commit is based on a partial revert of d143b3d1cd89 ("rcu: Simplify rcu_read_unlock_special() deferred wakeups") with an additional in_irq() check added. Signed-off-by: Joel Fernandes (Google) <j...@joelfernandes.org> --- v1->v2: Some minor character encoding issues in changelog corrected. Note that I am still testing this patch, but I sent an early RFC for your feedback. Thanks! kernel/rcu/tree_plugin.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 2defc7fe74c3..f4b3055026dc 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -621,6 +621,11 @@ static void rcu_read_unlock_special(struct task_struct *t) // Using softirq, safe to awaken, and we get // no help from enabling irqs, unlike bh/preempt. raise_softirq_irqoff(RCU_SOFTIRQ); + } else if (exp && in_irq() && !use_softirq && + !t->rcu_read_unlock_special.b.deferred_qs) { + // Safe to awaken rcuc kthread which will be + // scheduled in from the interrupt return path. + invoke_rcu_core(); } else { // Enabling BH or preempt does reschedule, so... // Also if no expediting or NO_HZ_FULL, slow is OK. -- 2.23.0.rc1.153.gdeed80330f-goog