[tip: sched/core] entry: Explicitly flush pending rcuog wakeup before last rescheduling point
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 47b8ff194c1fd73d58dc339b597d466fe48c8958
Gitweb:
https://git.kernel.org/tip/47b8ff194c1fd73d58dc339b597d466fe48c8958
Author:Frederic Weisbecker
AuthorDate:Mon, 01 Feb 2021 00:05:47 +01:00
Committer: Ingo Molnar
CommitterDate: Wed, 17 Feb 2021 14:12:43 +01:00
entry: Explicitly flush pending rcuog wakeup before last rescheduling point
Following the idle loop model, cleanly check for pending rcuog wakeup
before the last rescheduling point on resuming to user mode. This
way we can avoid to do it from rcu_user_enter() with the last resort
self-IPI hack that enforces rescheduling.
Signed-off-by: Frederic Weisbecker
Signed-off-by: Peter Zijlstra (Intel)
Signed-off-by: Ingo Molnar
Cc: sta...@vger.kernel.org
Link: https://lkml.kernel.org/r/20210131230548.32970-5-frede...@kernel.org
---
kernel/entry/common.c | 7 +++
kernel/rcu/tree.c | 12 +++-
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/kernel/entry/common.c b/kernel/entry/common.c
index f09cae3..8442e5c 100644
--- a/kernel/entry/common.c
+++ b/kernel/entry/common.c
@@ -184,6 +184,10 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs
*regs,
* enabled above.
*/
local_irq_disable_exit_to_user();
+
+ /* Check if any of the above work has queued a deferred wakeup
*/
+ rcu_nocb_flush_deferred_wakeup();
+
ti_work = READ_ONCE(current_thread_info()->flags);
}
@@ -197,6 +201,9 @@ static void exit_to_user_mode_prepare(struct pt_regs *regs)
lockdep_assert_irqs_disabled();
+ /* Flush pending rcuog wakeup before the last need_resched() check */
+ rcu_nocb_flush_deferred_wakeup();
+
if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK))
ti_work = exit_to_user_mode_loop(regs, ti_work);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 4b1e5bd..2ebc211 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -707,13 +707,15 @@ noinstr void rcu_user_enter(void)
lockdep_assert_irqs_disabled();
/*
-* We may be past the last rescheduling opportunity in the entry code.
-* Trigger a self IPI that will fire and reschedule once we resume to
-* user/guest mode.
+* Other than generic entry implementation, we may be past the last
+* rescheduling opportunity in the entry code. Trigger a self IPI
+* that will fire and reschedule once we resume in user/guest mode.
*/
instrumentation_begin();
- if (do_nocb_deferred_wakeup(rdp) && need_resched())
- irq_work_queue(this_cpu_ptr(&late_wakeup_work));
+ if (!IS_ENABLED(CONFIG_GENERIC_ENTRY) || (current->flags & PF_VCPU)) {
+ if (do_nocb_deferred_wakeup(rdp) && need_resched())
+ irq_work_queue(this_cpu_ptr(&late_wakeup_work));
+ }
instrumentation_end();
rcu_eqs_enter(true);
[tip: sched/core] entry: Explicitly flush pending rcuog wakeup before last rescheduling point
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2c910e0753dc424dfdeb1f8e230ad8f187a744a7
Gitweb:
https://git.kernel.org/tip/2c910e0753dc424dfdeb1f8e230ad8f187a744a7
Author:Frederic Weisbecker
AuthorDate:Mon, 01 Feb 2021 00:05:47 +01:00
Committer: Peter Zijlstra
CommitterDate: Wed, 10 Feb 2021 14:44:51 +01:00
entry: Explicitly flush pending rcuog wakeup before last rescheduling point
Following the idle loop model, cleanly check for pending rcuog wakeup
before the last rescheduling point on resuming to user mode. This
way we can avoid to do it from rcu_user_enter() with the last resort
self-IPI hack that enforces rescheduling.
Signed-off-by: Frederic Weisbecker
Signed-off-by: Peter Zijlstra (Intel)
Cc: sta...@vger.kernel.org
Link: https://lkml.kernel.org/r/20210131230548.32970-5-frede...@kernel.org
---
kernel/entry/common.c | 7 +++
kernel/rcu/tree.c | 12 +++-
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/kernel/entry/common.c b/kernel/entry/common.c
index f09cae3..8442e5c 100644
--- a/kernel/entry/common.c
+++ b/kernel/entry/common.c
@@ -184,6 +184,10 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs
*regs,
* enabled above.
*/
local_irq_disable_exit_to_user();
+
+ /* Check if any of the above work has queued a deferred wakeup
*/
+ rcu_nocb_flush_deferred_wakeup();
+
ti_work = READ_ONCE(current_thread_info()->flags);
}
@@ -197,6 +201,9 @@ static void exit_to_user_mode_prepare(struct pt_regs *regs)
lockdep_assert_irqs_disabled();
+ /* Flush pending rcuog wakeup before the last need_resched() check */
+ rcu_nocb_flush_deferred_wakeup();
+
if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK))
ti_work = exit_to_user_mode_loop(regs, ti_work);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 4b1e5bd..2ebc211 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -707,13 +707,15 @@ noinstr void rcu_user_enter(void)
lockdep_assert_irqs_disabled();
/*
-* We may be past the last rescheduling opportunity in the entry code.
-* Trigger a self IPI that will fire and reschedule once we resume to
-* user/guest mode.
+* Other than generic entry implementation, we may be past the last
+* rescheduling opportunity in the entry code. Trigger a self IPI
+* that will fire and reschedule once we resume in user/guest mode.
*/
instrumentation_begin();
- if (do_nocb_deferred_wakeup(rdp) && need_resched())
- irq_work_queue(this_cpu_ptr(&late_wakeup_work));
+ if (!IS_ENABLED(CONFIG_GENERIC_ENTRY) || (current->flags & PF_VCPU)) {
+ if (do_nocb_deferred_wakeup(rdp) && need_resched())
+ irq_work_queue(this_cpu_ptr(&late_wakeup_work));
+ }
instrumentation_end();
rcu_eqs_enter(true);
