[ https://issues.apache.org/jira/browse/IMPALA-9262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sahil Takiar resolved IMPALA-9262. ---------------------------------- Resolution: Fixed Issue didn't occur over the weekend (it was happening almost every day before the most recent fix). > Blacklist test test_kill_impalad_with_running_queries fails in exhaustive mode > ------------------------------------------------------------------------------ > > Key: IMPALA-9262 > URL: https://issues.apache.org/jira/browse/IMPALA-9262 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec > Affects Versions: Impala 3.4.0 > Reporter: Laszlo Gaal > Assignee: Sahil Takiar > Priority: Blocker > Labels: broken-build > Fix For: Impala 3.4.0 > > > Test proc test_kill_impalad_with_running_queries in test_blacklist.py failed > in multiple builds in executive mode with identical symptoms: > Backtrace is > {code} > custom_cluster/test_blacklist.py:145: in > test_kill_impalad_with_running_queries > assert "TransmitData() to " in str(e) or "EndDataStream() to " in str(e) > E assert ('TransmitData() to ' in 'Query was expected to fail\nassert > False' or 'EndDataStream() to ' in 'Query was expected to fail\nassert False') > E + where 'Query was expected to fail\nassert False' = > str(AssertionError(u'Query was expected to fail\nassert False',)) > E + and 'Query was expected to fail\nassert False' = > str(AssertionError(u'Query was expected to fail\nassert False',)) > {code} > Standard error is: > {code} > 02:19:19 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) > 02:19:19 MainThread: Starting State Store logging to > /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/statestored.INFO > 02:19:19 MainThread: Starting Catalog Service logging to > /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/catalogd.INFO > 02:19:20 MainThread: Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad.INFO > 02:19:20 MainThread: Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO > 02:19:20 MainThread: Starting Impala Daemon logging to > /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO > 02:19:23 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 02:19:23 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 02:19:23 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25000 > 02:19:23 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 02:19:24 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 02:19:24 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25000 > 02:19:24 MainThread: Waiting for num_known_live_backends=3. Current value: 2 > 02:19:25 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 02:19:25 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25000 > 02:19:25 MainThread: num_known_live_backends has reached value: 3 > 02:19:25 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 02:19:25 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25001 > 02:19:25 MainThread: num_known_live_backends has reached value: 3 > 02:19:25 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) > 02:19:25 MainThread: Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25002 > 02:19:25 MainThread: num_known_live_backends has reached value: 3 > 02:19:25 MainThread: Impala Cluster Running with 3 nodes (1 coordinators, 2 > executors). > DEBUG:impala_cluster:Found 3 impalad/1 statestored/1 catalogd process(es) > INFO:impala_service:Getting metric: statestore.live-backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25010 > INFO:impala_service:Metric 'statestore.live-backends' has reached desired > value: 4 > DEBUG:impala_service:Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25000 > INFO:impala_service:num_known_live_backends has reached value: 3 > DEBUG:impala_service:Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25001 > INFO:impala_service:num_known_live_backends has reached value: 3 > DEBUG:impala_service:Getting num_known_live_backends from > impala-ec2-centos74-m5-4xlarge-ondemand-1066.vpc.cloudera.com:25002 > INFO:impala_service:num_known_live_backends has reached value: 3 > ERROR:impala.hiveserver2:Failed to open transport (tries_left=3) > Traceback (most recent call last): > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py", > line 1009, in _execute > return func(request) > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", > line 176, in OpenSession > return self.recv_OpenSession() > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", > line 188, in recv_OpenSession > (fname, mtype, rseqid) = iprot.readMessageBegin() > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", > line 126, in readMessageBegin > sz = self.readI32() > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", > line 206, in readI32 > buff = self.trans.readAll(4) > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", > line 58, in readAll > chunk = self.read(sz - have) > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py", > line 159, in read > self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size))) > File > "/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py", > line 120, in read > message='TSocket read 0 bytes') > TTransportException: TSocket read 0 bytes > INFO:impala_cluster:Killing <ImpaladProcess PID: 23167 > (/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/build/latest/service/impalad > -is_coordinator=false -disconnected_session_timeout 3600 > -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost > -mem_limit=12884901888 -logbufsecs=5 -v=1 -max_log_files=0 > -log_filename=impalad_node2 > -log_dir=/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests > -beeswax_port=21002 -hs2_port=21052 -hs2_http_port=28002 -be_port=22002 > -krpc_port=27002 -state_store_subscriber_port=23002 -webserver_port=25002 > --default_query_options=)> with signal 9 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)