Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22254 )
Change subject: IMPALA-13253: Add option to enable keepalive for client connections ...................................................................... IMPALA-13253: Add option to enable keepalive for client connections Client connections can drop without an explicit close. This can happen if client machine resets or there is a network disruption. Some load balancers have an idle timeout that result in the connection becoming invalid without an explicit teardown. With short idle timeouts (e.g. AWS LB has a timeout of 350 seconds), this can impact many connections. This adds startup options to enable / tune TCP keepalive settings for client connections: client_keepalive_probe_period_s - idle time before doing keepalive probes If set to > 0, keepalive is enabled. client_keepalive_retry_period_s - time between keepalive probes client_keepalive_retry_count - number of keepalive probes These startup options mirror the startup options for Kudu's equivalent functionality. Thrift has preexisting support for turning on keepalive, but that support uses the OS defaults for keepalive settings. To add the ability to tune the keepalive settings, this implements a wrapper around the Thrift socket (both TLS and non-TLS) and manually sets the keepalive options on the socket (mirroring code from Kudu's Socket::SetTcpKeepAlive). This does not enable keepalive by default to make it easy to backport. A separate patch will turn keepalive on by default. Testing: - Added a custom cluster test that connects with impala-shell and verifies that the socket has the keepalive timer. Verified that it works on Ubuntu 20, Centos 7, and Redhat 8. - Used iptables to manually test cases where the client is unreachable and verified that the server detects that and closes the connection. Change-Id: I9e50f263006c456bc0797b8306aa4065e9713450 Reviewed-on: http://gerrit.cloudera.org:8080/22254 Reviewed-by: Yida Wu <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/service/impala-server.cc A tests/custom_cluster/test_client_keepalive.py 6 files changed, 271 insertions(+), 4 deletions(-) Approvals: Yida Wu: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/22254 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9e50f263006c456bc0797b8306aa4065e9713450 Gerrit-Change-Number: 22254 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
