[jira] [Created] (IMPALA-12114) SSL Thrift connections disconnect if idle more than ~150 seconds

[Joe McDonnell (Jira)]

Joe McDonnell created IMPALA-12114:
--------------------------------------

             Summary: SSL Thrift connections disconnect if idle more than ~150 
seconds
                 Key: IMPALA-12114
                 URL: https://issues.apache.org/jira/browse/IMPALA-12114
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 4.3.0
            Reporter: Joe McDonnell


A test cluster ran into issues with idle connections being disconnected when 
using SSL.

This reproduces on my development environment with these steps:
 # Start Impala with SSL enabled

{noformat}
bin/start-impala-cluster.py 
--impalad_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem
 --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem 
--ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem 
--hostname=localhost --idle_client_poll_period_s=30 -v=2" 
--state_store_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem
 --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem 
--ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem 
--hostname=localhost" 
--catalogd_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem
 --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem 
--ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem 
--hostname=localhost" --cluster_size=1{noformat}

 # Connect with impala-shell

{noformat}
impala-shell --ssl{noformat}

 # Leave this idle for 150+ seconds
In the Impalad logs will be a statement like this:

{noformat}
I0503 22:11:53.233147 206554 impala-server.cc:2488] Connection 
20470cb275a1d256:3d68601942f3179f from client 172.27.100.70:42540 to server 
hiveserver2-frontend closed. The connection had 2 associated 
session(s).{noformat}

 # Run a statement in impala-shell and will show that it needs to reconnect

{noformat}
default> show tables;
Caught exception TSocket read 0 bytes, type=<class 
'thrift.transport.TTransport.TTransportException'> in PingImpalaHS2Service. 
Caught exception [Errno 32] Broken pipe, type=<class 'socket.error'> in 
CloseSession. 
Warning: close session RPC failed: [Errno 32] Broken pipe, <class 
'socket.error'>
Connection lost, reconnecting...
... then it retries and succeeds{noformat}

Tracing through the code, it appears that this peek() call returns false:
{noformat}
      try {
        bytes_pending = input_->getTransport()->peek();
        break;
      } catch (const TTransportException& ttx) {{noformat}
bytes_pending is false, and this causes the connection to be closed.

This doesn't seem to impact Impala with older Thrift versions, so maybe 
something changed in Thrift 0.16.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)