Binglin Chang has submitted this change and it was merged. Change subject: rpc: LIFO service queue ......................................................................
rpc: LIFO service queue This adds a service queue implementation which differs from the previous one in a few ways: 1) when the worker threads want to dequeue work and there is none available, they push themselves onto a stack and wait for work on their own independent condition variable rather than a central one. Because it uses a stack, the wake-up behavior is LIFO rather than FIFO, which is preferable to reduce context switches and increase cache locality (hot threads stay hot). 2) when enqueueing a call, if there is already a waiting worker thread, then the call is assigned directly to that thread without going through the queue implementation. So, even if the queue implementation is heavy-weight (such as our priority queue) it does not affect the performance of workloads which have enough worker threads allocated. This improves rpc-bench throughput fairly substantially (~47% for sync, 66% for async): ./bin/rpc-bench -server_reactors 24 -async_call_concurrency 100 -worker_threads 40 -client_threads 24 -run_seconds 10 original: ------------------------ Mode: Sync Client threads: 24 Worker threads: 40 Server reactors: 24 ---------------------------------- Reqs/sec: 189006 User CPU per req: 21.9328us Sys CPU per req: 31.633us Ctx Sw. per req: 5.08355 Mode: Async Client reactors: 24 Call concurrency: 100 Worker threads: 40 Server reactors: 24 ---------------------------------- Reqs/sec: 202874 User CPU per req: 14.2749us Sys CPU per req: 23.538us Ctx Sw. per req: 2.7318 lifo queue without hash change: ----------------------- Reqs/sec: 278730 User CPU per req: 19.8051us Sys CPU per req: 21.9387us Ctx Sw. per req: 4.04089 Mode: Async Client reactors: 24 Call concurrency: 100 Worker threads: 40 Server reactors: 24 ---------------------------------- Reqs/sec: 336000 User CPU per req: 11.0189us Sys CPU per req: 13.6012us Ctx Sw. per req: 1.48036 Change-Id: Ibd1eb677dd52f89683eb648b42918fcf51437215 Reviewed-on: http://gerrit.cloudera.org:8080/2938 Tested-by: Kudu Jenkins Reviewed-by: Binglin Chang <[email protected]> --- M build-support/tsan-suppressions.txt M src/kudu/rpc/CMakeLists.txt M src/kudu/rpc/service_pool.cc M src/kudu/rpc/service_pool.h A src/kudu/rpc/service_queue-test.cc A src/kudu/rpc/service_queue.cc M src/kudu/rpc/service_queue.h 7 files changed, 408 insertions(+), 84 deletions(-) Approvals: Binglin Chang: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/2938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ibd1eb677dd52f89683eb648b42918fcf51437215 Gerrit-PatchSet: 7 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Binglin Chang <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]>
