Sailesh Mukil has uploaded a new patch set (#3). Change subject: IMPALA-4456: Change query_exec_state_lock_ to a reader-writer lock ......................................................................
IMPALA-4456: Change query_exec_state_lock_ to a reader-writer lock This patch includes a ported reader-writer lock implementation from Google utils and is used to replace the mutex query_exec_state_lock_ with a RW lock. The main offender, ReportExecStatus() contends heavily for this lock on the coordinator for each update that a fragment instance sends as a RPC. This causes the creation of a larger number of new client connections on that node as a lot of connections are blocked on the ReportExecStatus() RPC. On running a highly concurrent workload, the number of clients reduced from ~1500 to ~1100 clients per node. Note that this may not seem like a lot just as in terms of clients, however, the number of clients created as a result of ReportExecStatus() has come down drastically. Due to this other long running RPCs like TransmitData() have ended up creating more clients. This also reduces lock contention which results in client calls to complete faster. Please refer the JIRA for more information. Future work to build on this patch to help with scalability: -IMPALA-4459: Consider making ReportExecStatus() RPC execute asynchronously -IMPALA-3977: TransmitData() should not block Change-Id: I790a95e7179f07aa7ba188d5422c5e054353ba0b --- M be/src/gutil/CMakeLists.txt A be/src/gutil/read_write_lock.cc A be/src/gutil/read_write_lock.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h A be/src/util/rw_lock.h 9 files changed, 178 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/5021/3 -- To view, visit http://gerrit.cloudera.org:8080/5021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I790a95e7179f07aa7ba188d5422c5e054353ba0b Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Sailesh Mukil <sail...@cloudera.com> Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>