Henry Robinson has submitted this change and it was merged. Change subject: IMPALA-4135: Thrift threaded server times-out connections during high load ......................................................................
IMPALA-4135: Thrift threaded server times-out connections during high load During times of high load, Thrift's TThreadedServer can't keep up with the rate of new socket connections, causing some to time out. This patch creates TAcceptQueueServer, which is a modified version of TThreadedServer that calls accept() and then hands the returned TTransport off to a thread pool to handle setting up the connection. This ensures that accept() is called as quickly as possible, preventing connections from timing out while waiting. It also adds a metric, connection-setup-queue-size, to monitor the number of accepted connections waiting to be processed. A flag, --accepted_cnxn_queue_depth, controls the size of the accepted connection buffer. Testing: - New test added to thrift-server-test. (Disabled by default, due to high ulimit -n requirement) - Locally with the repro shown in IMPALA-4135. - On the 16-node with a real repro query. - Ran the stress test for a while. Change-Id: Ie50e728974ef31a9d49132a0b3f7cde2a4f3356d Reviewed-on: http://gerrit.cloudera.org:8080/4519 Tested-by: Internal Jenkins Reviewed-by: Henry Robinson <he...@cloudera.com> --- M be/src/common/global-flags.cc M be/src/rpc/CMakeLists.txt A be/src/rpc/TAcceptQueueServer.cpp A be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M common/thrift/metrics.json 8 files changed, 545 insertions(+), 14 deletions(-) Approvals: Henry Robinson: Looks good to me, approved Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/4519 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ie50e728974ef31a9d49132a0b3f7cde2a4f3356d Gerrit-PatchSet: 9 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall <tmarsh...@cloudera.com> Gerrit-Reviewer: Henry Robinson <he...@cloudera.com> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Juan Yu <j...@cloudera.com> Gerrit-Reviewer: Matthew Jacobs <m...@cloudera.com> Gerrit-Reviewer: Sailesh Mukil <sail...@cloudera.com> Gerrit-Reviewer: Thomas Tauber-Marshall <tmarsh...@cloudera.com>
