[PATCH 2/2] locking/rwsem: Don't hog RCU read lock while optimistically spinning

[Sultan Alsawaf]

From: Sultan Alsawaf <sul...@kerneltoast.com>

There's no reason to hold an RCU read lock the entire time while
optimistically spinning for a rwsem. This can needlessly lengthen RCU
grace periods and slow down synchronize_rcu() when it doesn't brute
force the RCU grace period via rcupdate.rcu_expedited=1.

Signed-off-by: Sultan Alsawaf <sul...@kerneltoast.com>
---
 kernel/locking/rwsem.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
index f11b9bd3431d..a1e3ceb254d1 100644
--- a/kernel/locking/rwsem.c
+++ b/kernel/locking/rwsem.c
@@ -723,8 +723,10 @@ rwsem_spin_on_owner(struct rw_semaphore *sem, unsigned 
long nonspinnable)
        if (state != OWNER_WRITER)
                return state;
 
-       rcu_read_lock();
        for (;;) {
+               bool same_owner;
+
+               rcu_read_lock();
                /*
                 * When a waiting writer set the handoff flag, it may spin
                 * on the owner as well. Once that writer acquires the lock,
@@ -732,27 +734,32 @@ rwsem_spin_on_owner(struct rw_semaphore *sem, unsigned 
long nonspinnable)
                 * handoff bit is set.
                 */
                new = rwsem_owner_flags(sem, &new_flags);
-               if ((new != owner) || (new_flags != flags)) {
-                       state = rwsem_owner_state(new, new_flags, nonspinnable);
-                       break;
-               }
 
                /*
-                * Ensure we emit the owner->on_cpu, dereference _after_
-                * checking sem->owner still matches owner, if that fails,
+                * Ensure sem->owner still matches owner. If that fails,
                 * owner might point to free()d memory, if it still matches,
                 * the rcu_read_lock() ensures the memory stays valid.
                 */
-               barrier();
+               same_owner = new == owner && new_flags == flags;
+               if (same_owner && !owner_on_cpu(owner))
+                       state = OWNER_NONSPINNABLE;
+               rcu_read_unlock();
 
-               if (need_resched() || !owner_on_cpu(owner)) {
+               if (!same_owner) {
+                       state = rwsem_owner_state(new, new_flags, nonspinnable);
+                       break;
+               }
+
+               if (state == OWNER_NONSPINNABLE)
+                       break;
+
+               if (need_resched()) {
                        state = OWNER_NONSPINNABLE;
                        break;
                }
 
                cpu_relax();
        }
-       rcu_read_unlock();
 
        return state;
 }
-- 
2.28.0