Joe McDonnell created IMPALA-12114: -------------------------------------- Summary: SSL Thrift connections disconnect if idle more than ~150 seconds Key: IMPALA-12114 URL: https://issues.apache.org/jira/browse/IMPALA-12114 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 4.3.0 Reporter: Joe McDonnell
A test cluster ran into issues with idle connections being disconnected when using SSL. This reproduces on my development environment with these steps: # Start Impala with SSL enabled {noformat} bin/start-impala-cluster.py --impalad_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost --idle_client_poll_period_s=30 -v=2" --state_store_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost" --catalogd_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost" --cluster_size=1{noformat} # Connect with impala-shell {noformat} impala-shell --ssl{noformat} # Leave this idle for 150+ seconds In the Impalad logs will be a statement like this: {noformat} I0503 22:11:53.233147 206554 impala-server.cc:2488] Connection 20470cb275a1d256:3d68601942f3179f from client 172.27.100.70:42540 to server hiveserver2-frontend closed. The connection had 2 associated session(s).{noformat} # Run a statement in impala-shell and will show that it needs to reconnect {noformat} default> show tables; Caught exception TSocket read 0 bytes, type=<class 'thrift.transport.TTransport.TTransportException'> in PingImpalaHS2Service. Caught exception [Errno 32] Broken pipe, type=<class 'socket.error'> in CloseSession. Warning: close session RPC failed: [Errno 32] Broken pipe, <class 'socket.error'> Connection lost, reconnecting... ... then it retries and succeeds{noformat} Tracing through the code, it appears that this peek() call returns false: {noformat} try { bytes_pending = input_->getTransport()->peek(); break; } catch (const TTransportException& ttx) {{noformat} bytes_pending is false, and this causes the connection to be closed. This doesn't seem to impact Impala with older Thrift versions, so maybe something changed in Thrift 0.16. -- This message was sent by Atlassian Jira (v8.20.10#820010)