[
https://issues.apache.org/jira/browse/IMPALA-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953244#comment-16953244
]
Sahil Takiar commented on IMPALA-4238:
--------------------------------------
For whatever reason, I've seen this happen on S3 occasionally as well. Might be
worth disabling for S3 builds since these tests should be agnostic of the
filesystem. I don't think they even run queries that use S3.
> custom_cluster/test_client_ssl.py TestClientSsl.test_ssl AssertionError:
> SIGINT was not caught by shell within 30s
> ------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-4238
> URL: https://issues.apache.org/jira/browse/IMPALA-4238
> Project: IMPALA
> Issue Type: Bug
> Components: Security
> Affects Versions: Impala 2.8.0, Impala 2.10.0
> Reporter: Harrison Sheinblatt
> Assignee: Dinesh Garg
> Priority: Major
> Labels: broken-build, flaky
>
> asf master core test failure:
> http://sandbox.jenkins.sf.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core/540/
> http://sandbox.jenkins.sf.cloudera.com/job/impala-umbrella-build-and-test/4921/console
> {noformat}
> 08:18:51 =================================== FAILURES
> ===================================
> 08:18:51 TestClientSsl.test_ssl[exec_option: {'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0,
> 'num_nodes': 0} | table_format: text/none]
> 08:18:51
> 08:18:51 self = <test_client_ssl.TestClientSsl object at 0x4b40810>
> 08:18:51 vector = <tests.common.test_vector.TestVector object at 0x4ac9110>
> 08:18:51
> 08:18:51 @pytest.mark.execute_serially
> 08:18:51
> @CustomClusterTestSuite.with_args("--ssl_server_certificate=%s/server-cert.pem
> "
> 08:18:51
> "--ssl_private_key=%s/server-key.pem"
> 08:18:51 % (CERT_DIR, CERT_DIR))
> 08:18:51 def test_ssl(self, vector):
> 08:18:51
> 08:18:51 self._verify_negative_cases()
> 08:18:51 # TODO: This is really two different tests, but the custom
> cluster takes too long to
> 08:18:51 # start. Make it so that custom clusters can be specified
> across test suites.
> 08:18:51 self._validate_positive_cases("%s/server-cert.pem" %
> self.CERT_DIR)
> 08:18:51
> 08:18:51 # No certificate checking: will accept any cert.
> 08:18:51 self._validate_positive_cases()
> 08:18:51
> 08:18:51 # Test cancelling a query
> 08:18:51 impalad = ImpaladService(socket.getfqdn())
> 08:18:51 impalad.wait_for_num_in_flight_queries(0)
> 08:18:51 p = ImpalaShell(args="--ssl")
> 08:18:51 p.send_cmd("SET DEBUG_ACTION=0:OPEN:WAIT")
> 08:18:51 p.send_cmd("select count(*) from functional.alltypes")
> 08:18:51 impalad.wait_for_num_in_flight_queries(1)
> 08:18:51
> 08:18:51 LOG = logging.getLogger('test_client_ssl')
> 08:18:51 LOG.info("Cancelling query")
> 08:18:51 num_tries = 0
> 08:18:51 # In practice, sending SIGINT to the shell process doesn't
> always seem to get caught
> 08:18:51 # (and a search shows up some bugs in Python where SIGINT
> might be ignored). So retry
> 08:18:51 # for 30s until one signal takes.
> 08:18:51 while impalad.get_num_in_flight_queries() == 1:
> 08:18:51 time.sleep(1)
> 08:18:51 LOG.info("Sending signal...")
> 08:18:51 os.kill(p.pid(), signal.SIGINT)
> 08:18:51 num_tries += 1
> 08:18:51 > assert num_tries < 30, "SIGINT was not caught by shell
> within 30s"
> 08:18:51 E AssertionError: SIGINT was not caught by shell within 30s
> 08:18:51 E assert 30 < 30
> 08:18:51
> 08:18:51 custom_cluster/test_client_ssl.py:85: AssertionError
> 08:18:51 ---------------------------- Captured stdout setup
> -----------------------------
> 08:18:51 Starting State Store logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 08:18:51 Starting Catalog Service logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 08:18:51 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 08:18:51 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 08:18:51 Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
> 08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
> 08:18:51 Waiting for Catalog... Status: 53 DBs / 1077 tables (ready=True)
> 08:18:51 Impala Cluster Running with 3 nodes.
> 08:18:51 ---------------------------- Captured stderr setup
> -----------------------------
> 08:18:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
> 08:18:51 MainThread: Debug webpage not yet available.
> 08:18:51 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
> 08:18:51 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
> 08:18:51 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
> 08:18:51 MainThread: num_known_live_backends has reached value: 3
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25001
> 08:18:51 MainThread: num_known_live_backends has reached value: 3
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25002
> 08:18:51 MainThread: num_known_live_backends has reached value: 3
> 08:18:51 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 08:18:51 MainThread: Getting metric: statestore.live-backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25010
> 08:18:51 MainThread: Metric 'statestore.live-backends' has reach desired
> value: 4
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25000
> 08:18:51 MainThread: num_known_live_backends has reached value: 3
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25001
> 08:18:51 MainThread: num_known_live_backends has reached value: 3
> 08:18:51 MainThread: Getting num_known_live_backends from
> impala-boost-static-burst-slave-07f9.vpc.cloudera.com:25002
> 08:18:51 MainThread: num_known_live_backends has reached value: 3
> 08:18:51 -- connecting to: localhost:21000
> 08:18:51 ----------------------------- Captured stderr call
> -----------------------------
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Cancelling query
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 MainThread: Getting num_in_flight_queries from
> localhost.localdomain:25000
> 08:18:51 MainThread: Sending signal...
> 08:18:51 generated xml file:
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/custom_cluster_tests/results/TEST-impala-custom-cluster.xml
>
> 08:18:51 == 1 failed, 42 passed, 20 skipped, 6 xfailed, 2 warnings in 3094.24
> seconds ===
> 08:18:51 Error in
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/tests/run-custom-cluster-tests.sh
> at line 42: impala-py.test custom_cluster/ authorization/ ${AUX_CUSTOM_DIR} \
> 08:18:51 Error in
> /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/buildall.sh
> at line 412: "${IMPALA_HOME}/bin/run-all-tests.sh" -e $EXPLORATION_STRATEGY
> 08:18:51 + echo 'buildall.sh ' -format '-snapshot_file
> /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh5-85-SNAPSHOT.tar.gz'
> '-metastore_snapshot_file
> /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/hive_metastore_snapshot/hive_impala_dump_cdh5-85.txt
> failed.'
> 08:18:51 buildall.sh -format -snapshot_file
> /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh5-85-SNAPSHOT.tar.gz
> -metastore_snapshot_file
> /data/jenkins/workspace/impala-umbrella-build-and-test/testdata/hive_metastore_snapshot/hive_impala_dump_cdh5-85.txt
> failed.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
