[jira] [Commented] (IMPALA-10050) DCHECK was hit possibly while executing TestFailpoints::test_failpoints
[ https://issues.apache.org/jira/browse/IMPALA-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172412#comment-17172412 ] Attila Jeges commented on IMPALA-10050: --- [~rizaon] randomly assigning to you. Please feel free to reassign. > DCHECK was hit possibly while executing TestFailpoints::test_failpoints > --- > > Key: IMPALA-10050 > URL: https://issues.apache.org/jira/browse/IMPALA-10050 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Attila Jeges >Assignee: Riza Suminto >Priority: Blocker > Labels: broken-build, crash > Fix For: Impala 4.0 > > > A DCHECK was hit during ASAN core e2e tests. Time-frame suggests that it > happened while executing TestFailpoints::test_failpoints e2e test. > {code} > 10:56:38 TestFailpoints.test_failpoints[protocol: beeswax | table_format: > avro/snap/block | exec_option: {'batch_size': 0, 'num_nodes': 0, > 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | mt_dop: 4 | > location: PREPARE | action: MEM_LIMIT_EXCEEDED | query: select 1 from > alltypessmall a join alltypessmall b on a.id = b.id] > 10:56:38 failure/test_failpoints.py:128: in test_failpoints > 10:56:38 self.execute_query(query, vector.get_value('exec_option')) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:811: > in wrapper > 10:56:38 return function(*args, **kwargs) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:843: > in execute_query > 10:56:38 return self.__execute_query(self.client, query, query_options) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:909: > in __execute_query > 10:56:38 return impalad_client.execute(query, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_connection.py:205: > in execute > 10:56:38 return self.__beeswax_client.execute(sql_stmt, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:187: > in execute > 10:56:38 handle = self.__execute_query(query_string.strip(), user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:365: > in __execute_query > 10:56:38 self.wait_for_finished(handle) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:386: > in wait_for_finished > 10:56:38 raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > 10:56:38 E ImpalaBeeswaxException: ImpalaBeeswaxException: > 10:56:38 EQuery aborted:RPC from 127.0.0.1:27000 to 127.0.0.1:27002 failed > 10:56:38 E TransmitData() to 127.0.0.1:27002 failed: Network error: Client > connection negotiation failed: client connection to 127.0.0.1:27002: connect: > Connection refused (error 111) > {code} > Impalad log: > {code} > Log file created at: 2020/08/05 01:52:56 > Running on machine: > impala-ec2-centos74-r5-4xlarge-ondemand-017c.vpc.cloudera.com > Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg > F0805 01:52:56.979769 17313 query-state.cc:803] > 3941a3d92a71e242:15c963f3] Check failed: is_cancelled_.Load() == 1 (0 > vs. 1) > {code} > Stack trace > {code} > Thread 368 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x0004 > rsi = 0x43a1 rdi = 0x37e4 > rbp = 0x7efcd4c53080 rsp = 0x7efcd4c52d08 > r8 = 0xr9 = 0x7efcd4c52b80 > r10 = 0x0008 r11 = 0x0206 > r12 = 0x093de7c0 r13 = 0x0086 > r14 = 0x093de7c4 r15 = 0x093d6de0 > rip = 0x7f05c9d231f7 > Found by: given as instruction pointer in context > 1 impalad!google::LogMessage::Flush() + 0x1eb > rbp = 0x7efcd4c53250 rsp = 0x7efcd4c53090 > rip = 0x05727e3b > Found by: previous frame's frame pointer > 2 impalad!google::LogMessageFatal::~LogMessageFatal() + 0x9 > rbx = 0x7efcd4c532a0 rbp = 0x7efcd4c53310 > rsp = 0x7efcd4c53130 r12 = 0x0fe01a982628 > r13 = 0x61d000da0a6c r14 = 0x7efcd4c53250 > r15 = 0x7efcd4c53270 rip = 0x0572ba39 > Found by: call frame info > 3 impalad!impala::QueryState::MonitorFInstances() [query-state.cc : 803 + > 0x45] > rbx = 0x7efcd4c532a0
[jira] [Commented] (IMPALA-10050) DCHECK was hit possibly while executing TestFailpoints::test_failpoints
[ https://issues.apache.org/jira/browse/IMPALA-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172571#comment-17172571 ] Thomas Tauber-Marshall commented on IMPALA-10050: - I suspect this may have been caused by https://gerrit.cloudera.org/#/c/16215/ so [~wzhou] might be the right person to take a look > DCHECK was hit possibly while executing TestFailpoints::test_failpoints > --- > > Key: IMPALA-10050 > URL: https://issues.apache.org/jira/browse/IMPALA-10050 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Attila Jeges >Assignee: Riza Suminto >Priority: Blocker > Labels: broken-build, crash, flaky > Fix For: Impala 4.0 > > > A DCHECK was hit during ASAN core e2e tests. Time-frame suggests that it > happened while executing TestFailpoints::test_failpoints e2e test. > {code} > 10:56:38 TestFailpoints.test_failpoints[protocol: beeswax | table_format: > avro/snap/block | exec_option: {'batch_size': 0, 'num_nodes': 0, > 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | mt_dop: 4 | > location: PREPARE | action: MEM_LIMIT_EXCEEDED | query: select 1 from > alltypessmall a join alltypessmall b on a.id = b.id] > 10:56:38 failure/test_failpoints.py:128: in test_failpoints > 10:56:38 self.execute_query(query, vector.get_value('exec_option')) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:811: > in wrapper > 10:56:38 return function(*args, **kwargs) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:843: > in execute_query > 10:56:38 return self.__execute_query(self.client, query, query_options) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:909: > in __execute_query > 10:56:38 return impalad_client.execute(query, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_connection.py:205: > in execute > 10:56:38 return self.__beeswax_client.execute(sql_stmt, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:187: > in execute > 10:56:38 handle = self.__execute_query(query_string.strip(), user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:365: > in __execute_query > 10:56:38 self.wait_for_finished(handle) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:386: > in wait_for_finished > 10:56:38 raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > 10:56:38 E ImpalaBeeswaxException: ImpalaBeeswaxException: > 10:56:38 EQuery aborted:RPC from 127.0.0.1:27000 to 127.0.0.1:27002 failed > 10:56:38 E TransmitData() to 127.0.0.1:27002 failed: Network error: Client > connection negotiation failed: client connection to 127.0.0.1:27002: connect: > Connection refused (error 111) > {code} > Impalad log: > {code} > Log file created at: 2020/08/05 01:52:56 > Running on machine: > impala-ec2-centos74-r5-4xlarge-ondemand-017c.vpc.cloudera.com > Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg > F0805 01:52:56.979769 17313 query-state.cc:803] > 3941a3d92a71e242:15c963f3] Check failed: is_cancelled_.Load() == 1 (0 > vs. 1) > {code} > Stack trace > {code} > Thread 368 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x0004 > rsi = 0x43a1 rdi = 0x37e4 > rbp = 0x7efcd4c53080 rsp = 0x7efcd4c52d08 > r8 = 0xr9 = 0x7efcd4c52b80 > r10 = 0x0008 r11 = 0x0206 > r12 = 0x093de7c0 r13 = 0x0086 > r14 = 0x093de7c4 r15 = 0x093d6de0 > rip = 0x7f05c9d231f7 > Found by: given as instruction pointer in context > 1 impalad!google::LogMessage::Flush() + 0x1eb > rbp = 0x7efcd4c53250 rsp = 0x7efcd4c53090 > rip = 0x05727e3b > Found by: previous frame's frame pointer > 2 impalad!google::LogMessageFatal::~LogMessageFatal() + 0x9 > rbx = 0x7efcd4c532a0 rbp = 0x7efcd4c53310 > rsp = 0x7efcd4c53130 r12 = 0x0fe01a982628 > r13 = 0x61d000da0a6c r14 = 0x7efcd4c53250 > r15 = 0x7efcd4c53270 rip = 0x0572ba39 > Found by: call frame info > 3 impalad!impala::QueryS
[jira] [Commented] (IMPALA-10050) DCHECK was hit possibly while executing TestFailpoints::test_failpoints
[ https://issues.apache.org/jira/browse/IMPALA-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172684#comment-17172684 ] Wenzhe Zhou commented on IMPALA-10050: -- *01:52:58* failure/test_failpoints.py::TestFailpoints::test_failpoints[protocol: beeswax | table_format: avro/snap/block | exec_option: \{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | mt_dop: 4 | location: PREPARE | action: MEM_LIMIT_EXCEEDED | query: select 1 from alltypessmall a join alltypessmall b on a.id = b.id] FAILED h3. Error Message ImpalaBeeswaxException: ImpalaBeeswaxException: Query aborted:RPC from 127.0.0.1:27000 to 127.0.0.1:27002 failed TransmitData() to 127.0.0.1:27002 failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27002: connect: Connection refused (error 111) h3. Stacktrace failure/test_failpoints.py:128: in test_failpoints self.execute_query(query, vector.get_value('exec_option')) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:811: in wrapper return function(*args, **kwargs) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:843: in execute_query return self.__execute_query(self.client, query, query_options) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:909: in __execute_query return impalad_client.execute(query, user=user) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:365: in __execute_query self.wait_for_finished(handle) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:386: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:RPC from 127.0.0.1:27000 to 127.0.0.1:27002 failed E TransmitData() to 127.0.0.1:27002 failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27002: connect: Connection refused (error 111) > DCHECK was hit possibly while executing TestFailpoints::test_failpoints > --- > > Key: IMPALA-10050 > URL: https://issues.apache.org/jira/browse/IMPALA-10050 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Attila Jeges >Assignee: Wenzhe Zhou >Priority: Blocker > Labels: broken-build, crash, flaky > Fix For: Impala 4.0 > > > A DCHECK was hit during ASAN core e2e tests. Time-frame suggests that it > happened while executing TestFailpoints::test_failpoints e2e test. > {code} > 10:56:38 TestFailpoints.test_failpoints[protocol: beeswax | table_format: > avro/snap/block | exec_option: {'batch_size': 0, 'num_nodes': 0, > 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | mt_dop: 4 | > location: PREPARE | action: MEM_LIMIT_EXCEEDED | query: select 1 from > alltypessmall a join alltypessmall b on a.id = b.id] > 10:56:38 failure/test_failpoints.py:128: in test_failpoints > 10:56:38 self.execute_query(query, vector.get_value('exec_option')) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:811: > in wrapper > 10:56:38 return function(*args, **kwargs) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:843: > in execute_query > 10:56:38 return self.__execute_query(self.client, query, query_options) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:909: > in __execute_query > 10:56:38 return impalad_client.execute(query, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_connection.py:205: > in execute > 10:56:38 return self.__beeswax_client.execute(sql_stmt, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:187: > in execute > 10:56:38 handle = self.__execute_query(query_string.strip(), user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan
[jira] [Commented] (IMPALA-10050) DCHECK was hit possibly while executing TestFailpoints::test_failpoints
[ https://issues.apache.org/jira/browse/IMPALA-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180309#comment-17180309 ] Wenzhe Zhou commented on IMPALA-10050: -- The issue could be re-produced by running following script for a few hours: #!/bin/bash set -euo pipefail for iter in `seq 1 100`; do ${IMPALA_HOME}/bin/impala-py.test tests/failure/test_failpoints.py::TestFailpoints::test_failpoints done Added more debug messages in the code, and found that when executor report status with error, it may receive response with status 'ok' occasionally. If the executor is in the terminal state and receive response 'ok' for report with error, it will not call "Cancel()" hence hit the DCHECK error. The root cause is the bug in coordinator, not in executor. The patch [https://gerrit.cloudera.org/#/c/16215/] only apply to executor so this issue was not introduced by the patch. Ran the above script with the code rolled back before patch [https://gerrit.cloudera.org/#/c/16215/], the issue still could be re-produced. Ran more tests with the script and confirmed that this issue was introduced by patch [https://gerrit.cloudera.org/#/c/16192/] ([IMPALA-6788|http://issues.apache.org/jira/browse/IMPALA-6788]: Abort ExecFInstance() RPC loop early after query failure). > DCHECK was hit possibly while executing TestFailpoints::test_failpoints > --- > > Key: IMPALA-10050 > URL: https://issues.apache.org/jira/browse/IMPALA-10050 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Attila Jeges >Assignee: Wenzhe Zhou >Priority: Blocker > Labels: broken-build, crash, flaky > Fix For: Impala 4.0 > > > A DCHECK was hit during ASAN core e2e tests. Time-frame suggests that it > happened while executing TestFailpoints::test_failpoints e2e test. > {code} > 10:56:38 TestFailpoints.test_failpoints[protocol: beeswax | table_format: > avro/snap/block | exec_option: {'batch_size': 0, 'num_nodes': 0, > 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | mt_dop: 4 | > location: PREPARE | action: MEM_LIMIT_EXCEEDED | query: select 1 from > alltypessmall a join alltypessmall b on a.id = b.id] > 10:56:38 failure/test_failpoints.py:128: in test_failpoints > 10:56:38 self.execute_query(query, vector.get_value('exec_option')) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:811: > in wrapper > 10:56:38 return function(*args, **kwargs) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:843: > in execute_query > 10:56:38 return self.__execute_query(self.client, query, query_options) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:909: > in __execute_query > 10:56:38 return impalad_client.execute(query, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_connection.py:205: > in execute > 10:56:38 return self.__beeswax_client.execute(sql_stmt, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:187: > in execute > 10:56:38 handle = self.__execute_query(query_string.strip(), user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:365: > in __execute_query > 10:56:38 self.wait_for_finished(handle) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:386: > in wait_for_finished > 10:56:38 raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > 10:56:38 E ImpalaBeeswaxException: ImpalaBeeswaxException: > 10:56:38 EQuery aborted:RPC from 127.0.0.1:27000 to 127.0.0.1:27002 failed > 10:56:38 E TransmitData() to 127.0.0.1:27002 failed: Network error: Client > connection negotiation failed: client connection to 127.0.0.1:27002: connect: > Connection refused (error 111) > {code} > Impalad log: > {code} > Log file created at: 2020/08/05 01:52:56 > Running on machine: > impala-ec2-centos74-r5-4xlarge-ondemand-017c.vpc.cloudera.com > Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg > F0805 01:52:56.979769 17313 query-state.cc:803] > 3941a3d92a71e242:15c963f3] Check failed: is_cancelled_.Load() == 1 (0 > vs. 1) > {code} > Stack trace > {code} > Thread 368 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x
[jira] [Commented] (IMPALA-10050) DCHECK was hit possibly while executing TestFailpoints::test_failpoints
[ https://issues.apache.org/jira/browse/IMPALA-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186680#comment-17186680 ] ASF subversion and git services commented on IMPALA-10050: -- Commit 3733c4cc2cfb78d7f13463fb1ee9e1c4560d4a3d in impala's branch refs/heads/master from wzhou-code [ https://gitbox.apache.org/repos/asf?p=impala.git;h=3733c4c ] IMPALA-10050: Fixed DCHECK error for backend in terminal state. Recent patch for IMPALA-6788 makes coordinator to cancel inflight query fragment instances when it receives failure report from one backend. It's possible the BackendState::Cancel() is called for one fragment instance before the first execution status report from its backend is received and processed by the coordinator. Since the status of BackendState is set as Cancelled after Cancel() is called, the execution of the fragment instance is treated as Done in such case so that the status report will NOT be processed. Hence the backend receives response OK from coordinator even it sent a report with execution error. This make backend hit DCHECK error if backend in the terminal state with error. This patch fixs the issue by making coordinator send CANCELLED status in the response of status report if the backend status is not ok and the execution status report is not applied. Testing: - The issue could be reproduced by running test_failpoints for about 20 iterations. Verified the fixing by running test_failpoints over 200 iterations without DCHECK failure. - Passed TestProcessFailures::test_kill_coordinator. - Psssed TestRPCException::test_state_report_error. - Passed exhaustive tests. Change-Id: Iba6a72f98c0f9299c22c58830ec5a643335b966a Reviewed-on: http://gerrit.cloudera.org:8080/16303 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins > DCHECK was hit possibly while executing TestFailpoints::test_failpoints > --- > > Key: IMPALA-10050 > URL: https://issues.apache.org/jira/browse/IMPALA-10050 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Attila Jeges >Assignee: Wenzhe Zhou >Priority: Blocker > Labels: broken-build, crash, flaky > Fix For: Impala 4.0 > > > A DCHECK was hit during ASAN core e2e tests. Time-frame suggests that it > happened while executing TestFailpoints::test_failpoints e2e test. > {code} > 10:56:38 TestFailpoints.test_failpoints[protocol: beeswax | table_format: > avro/snap/block | exec_option: {'batch_size': 0, 'num_nodes': 0, > 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | mt_dop: 4 | > location: PREPARE | action: MEM_LIMIT_EXCEEDED | query: select 1 from > alltypessmall a join alltypessmall b on a.id = b.id] > 10:56:38 failure/test_failpoints.py:128: in test_failpoints > 10:56:38 self.execute_query(query, vector.get_value('exec_option')) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:811: > in wrapper > 10:56:38 return function(*args, **kwargs) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:843: > in execute_query > 10:56:38 return self.__execute_query(self.client, query, query_options) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_test_suite.py:909: > in __execute_query > 10:56:38 return impalad_client.execute(query, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_connection.py:205: > in execute > 10:56:38 return self.__beeswax_client.execute(sql_stmt, user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:187: > in execute > 10:56:38 handle = self.__execute_query(query_string.strip(), user=user) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:365: > in __execute_query > 10:56:38 self.wait_for_finished(handle) > 10:56:38 > /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/beeswax/impala_beeswax.py:386: > in wait_for_finished > 10:56:38 raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > 10:56:38 E ImpalaBeeswaxException: ImpalaBeeswaxException: > 10:56:38 EQuery aborted:RPC from 127.0.0.1:27000 to 127.0.0.1:27002 failed > 10:56:38 E TransmitData() to 127.0.0.1:27002 failed: Network error: Client > connection negotiation failed: client connection to 127.0.0.1:27002: connect: > Connection refused (error 111) > {code} > Impalad log: > {code} > Log file crea