In rwsem_wake(), we can safely check the wait_list to see if waiters
are present without lock when there are spinners to fall back on in
case we miss a waiter. The advantage is that we can save a pair of
spin_lock/unlock calls when the wait_list is empty. This translates
to a reduction in latency and hence slightly better performance.
The raw_spin_trylock_irqsave() call is also being replaced by a
raw_spin_is_locked() to reduce the overhead of irqsave/irqrestore
especially when the trylock fails.
On a 2-socket 36-core x86-64 E5-2699 v3 system, a rwsem microbenchmark
was run with 36 locking threads doing 1 million writer lock/unlock
operations each, the resulting locking rates (avg of 3 runs) on a
4.11-rc1 based kernel were 4,923 Mop/s and 5,136 Mop/s without and
with the patch respectively. That was an increase of about 4%.
On the same system, a 36-thread fio direct IO random write test to
the same 2GB file on a XFS formatted ramdisk was run. The aggregated
bandwidth was 1564.4 MB/s and 1593.0 MB/s before and after the
patch. That was an increase of about 2%.
Signed-off-by: Waiman Long <long...@redhat.com>
---
kernel/locking/rwsem-xadd.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 34e727f..f31dd61b 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -587,7 +587,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
/*
* If a spinner is present, it is not necessary to do the wakeup.
- * Try to do wakeup only if the trylock succeeds to minimize
+ * Try to do wakeup only if the wait_lock is free to minimize
* spinlock contention which may introduce too much delay in the
* unlock operation.
*
@@ -595,7 +595,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
* --------------- -----------------------
* [S] osq_unlock() [L] osq
* MB RMB
- * [RmW] rwsem_try_write_lock() [RmW] spin_trylock(wait_lock)
+ * [RmW] rwsem_try_write_lock() [RmW] spin_is_locked(wait_lock)
*
* Here, it is important to make sure that there won't be a missed
* wakeup while the rwsem is free and the only spinning writer goes
@@ -611,12 +611,18 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
* state is consulted before reading the wait_lock.
*/
smp_rmb();
- if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags))
+
+ /*
+ * Normally checking wait_list without wait_lock isn't safe
+ * as we may miss an incoming waiter. With spinners present,
+ * however, we have someone to fall back on in case that
+ * happens.
+ */
+ if (list_empty(&sem->wait_list) ||
+ raw_spin_is_locked(&sem->wait_lock))
return sem;
- goto locked;
}
raw_spin_lock_irqsave(&sem->wait_lock, flags);
-locked:
if (!list_empty(&sem->wait_list))
__rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
--
1.8.3.1
