[
https://issues.apache.org/jira/browse/IMPALA-10889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609559#comment-17609559
]
Michael Smith commented on IMPALA-10889:
----------------------------------------
This looks like a duplicate of IMPALA-10877 as well.
> TestExecutorGroups.test_admission_control_with_multiple_coords is flaky
> -----------------------------------------------------------------------
>
> Key: IMPALA-10889
> URL: https://issues.apache.org/jira/browse/IMPALA-10889
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Labels: broken-build
>
> Failed test:
> custom_cluster.test_executor_groups.TestExecutorGroups.test_admission_control_with_multiple_coords
> Error Message
> {code}
> AssertionError: Metric admission-controller.agg-num-running.default-pool did
> not reach value 1 in 30s. Dumping debug webpages in JSON format... Dumped
> memz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/memz.json
> Dumped metrics JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/metrics.json
> Dumped queries JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/queries.json
> Dumped sessions JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/sessions.json
> Dumped threadz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/threadz.json
> Dumped rpcz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/rpcz.json
> Dumping minidumps for impalads/catalogds... Dumped minidump for Impalad PID
> 23807 Dumped minidump for Impalad PID 23810 Dumped minidump for Impalad PID
> 24897 Dumped minidump for Impalad PID 24900 Dumped minidump for Catalogd PID
> 23746{code}
> Stacktrace
> {code}
> custom_cluster/test_executor_groups.py:579: in
> test_admission_control_with_multiple_coords
> "admission-controller.agg-num-running.default-pool", 1, timeout=30)
> common/impala_service.py:143: in wait_for_metric_value
> self.__metric_timeout_assert(metric_name, expected_value, timeout)
> common/impala_service.py:210: in __metric_timeout_assert
> assert 0, assert_string
> E AssertionError: Metric admission-controller.agg-num-running.default-pool
> did not reach value 1 in 30s.
> E Dumping debug webpages in JSON format...
> E Dumped memz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/memz.json
> E Dumped metrics JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/metrics.json
> E Dumped queries JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/queries.json
> E Dumped sessions JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/sessions.json
> E Dumped threadz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/threadz.json
> E Dumped rpcz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/rpcz.json
> E Dumping minidumps for impalads/catalogds...
> E Dumped minidump for Impalad PID 23807
> E Dumped minidump for Impalad PID 23810
> E Dumped minidump for Impalad PID 24897
> E Dumped minidump for Impalad PID 24900
> E Dumped minidump for Catalogd PID 23746
> {code}
> Standard Error
> {code}
> -- 2021-08-26 05:46:20,996 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=1
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 --use_exclusive_coordinators '--impalad_args=
> -executor_groups=coordinator ' --impalad_args=--default_query_options=
> 05:46:21 MainThread: Starting impala cluster without executors
> 05:46:21 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 05:46:21 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 05:46:21 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 05:46:21 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 05:46:24 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 05:46:24 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 05:46:24 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:24 MainThread: 'backends'
> 05:46:24 MainThread: Waiting for num_known_live_backends=1. Current value:
> None
> 05:46:25 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 05:46:25 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:25 MainThread: Waiting for num_known_live_backends=1. Current value: 0
> 05:46:26 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 05:46:26 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:26 MainThread: Waiting for num_known_live_backends=1. Current value: 0
> 05:46:27 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
> 05:46:27 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:27 MainThread: num_known_live_backends has reached value: 1
> 05:46:28 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 0
> executors).
> -- 2021-08-26 05:46:28,337 DEBUG MainThread: Found 1 impalad/1
> statestored/1 catalogd process(es)
> -- 2021-08-26 05:46:28,337 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010
> -- 2021-08-26 05:46:28,341 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 2
> -- 2021-08-26 05:46:28,341 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:28,343 INFO MainThread: num_known_live_backends has
> reached value: 1
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-08-26 05:46:28,489 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-08-26 05:46:28,509 INFO MainThread: Closing active operation
> -- 2021-08-26 05:46:28,511 INFO MainThread: Adding a coordinator
> -- 2021-08-26 05:46:28,511 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=2 --num_coordinators=2
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 --use_exclusive_coordinators
> --impalad_args=-executor_groups=coordinator
> --impalad_args=-default_pool_mem_limit=100g
> --impalad_args=--default_query_options=
> 05:46:28 MainThread: Starting impala cluster without executors
> 05:46:29 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 05:46:29 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 05:46:29 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 05:46:29 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 05:46:29 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 05:46:32 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:32 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:32 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:32 MainThread: 'backends'
> 05:46:32 MainThread: Waiting for num_known_live_backends=2. Current value:
> None
> 05:46:33 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:33 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:33 MainThread: Waiting for num_known_live_backends=2. Current value: 0
> 05:46:34 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:34 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:34 MainThread: Waiting for num_known_live_backends=2. Current value: 0
> 05:46:35 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:35 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:35 MainThread: num_known_live_backends has reached value: 2
> 05:46:35 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:35 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
> 05:46:35 MainThread: num_known_live_backends has reached value: 2
> 05:46:36 MainThread: Impala Cluster Running with 2 nodes (2 coordinators, 0
> executors).
> -- 2021-08-26 05:46:36,408 DEBUG MainThread: Found 2 impalad/1
> statestored/1 catalogd process(es)
> -- 2021-08-26 05:46:36,409 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010
> -- 2021-08-26 05:46:36,412 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 3
> -- 2021-08-26 05:46:36,412 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:36,414 INFO MainThread: num_known_live_backends has
> reached value: 2
> -- 2021-08-26 05:46:36,414 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
> -- 2021-08-26 05:46:36,416 INFO MainThread: num_known_live_backends has
> reached value: 2
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
> -- connecting to: localhost:21001
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
> -- connecting to: localhost:21000
> -- 2021-08-26 05:46:36,417 ERROR MainThread: Failed to open transport
> (tries_left=3)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py",
> line 1099, in _execute
> return func(request)
> File
> "/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py",
> line 206, in CloseSession
> return self.recv_CloseSession()
> File
> "/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py",
> line 218, in recv_CloseSession
> (fname, mtype, rseqid) = iprot.readMessageBegin()
> File
> "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py",
> line 134, in readMessageBegin
> sz = self.readI32()
> File
> "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py",
> line 217, in readI32
> buff = self.trans.readAll(4)
> File
> "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TTransport.py",
> line 60, in readAll
> chunk = self.read(sz - have)
> File
> "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TTransport.py",
> line 162, in read
> self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
> File
> "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 132, in read
> message='TSocket read 0 bytes')
> TTransportException: TSocket read 0 bytes
> -- connecting to localhost:21050 with impyla
> -- 2021-08-26 05:46:36,563 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-08-26 05:46:36,585 INFO MainThread: Closing active operation
> -- 2021-08-26 05:46:36,586 INFO MainThread: Adding 2 executors to group
> default-pool-group1 with minimum size 2
> -- 2021-08-26 05:46:36,586 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=2 --num_coordinators=0
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 --add_executors --impalad_args=-admission_control_slots=2
> --impalad_args=-executor_groups=default-pool-group1:2
> --impalad_args=-mem_limit=4g --impalad_args=--default_query_options=
> 05:46:37 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 05:46:37 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 05:46:37 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node3.INFO
> 05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
> 05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
> 05:46:40 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> 05:46:40 MainThread: num_known_live_backends has reached value: 4
> 05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
> 05:46:40 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
> 05:46:40 MainThread: num_known_live_backends has reached value: 4
> 05:46:41 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
> 05:46:41 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25002
> 05:46:41 MainThread: num_known_live_backends has reached value: 4
> 05:46:41 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es)
> 05:46:41 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25003
> 05:46:41 MainThread: num_known_live_backends has reached value: 4
> 05:46:41 MainThread: Impala Cluster Running with 2 nodes (0 coordinators, 2
> executors).
> -- 2021-08-26 05:46:41,307 DEBUG MainThread: Found 4 impalad/1
> statestored/1 catalogd process(es)
> -- 2021-08-26 05:46:41,307 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010
> -- 2021-08-26 05:46:41,310 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 5
> -- 2021-08-26 05:46:41,310 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:41,312 INFO MainThread: num_known_live_backends has
> reached value: 4
> -- 2021-08-26 05:46:41,312 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001
> -- 2021-08-26 05:46:41,314 INFO MainThread: num_known_live_backends has
> reached value: 4
> -- 2021-08-26 05:46:41,314 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25002
> -- 2021-08-26 05:46:41,316 INFO MainThread: num_known_live_backends has
> reached value: 4
> -- 2021-08-26 05:46:41,316 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25003
> -- 2021-08-26 05:46:41,318 INFO MainThread: num_known_live_backends has
> reached value: 4
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
> SET mt_dop=2;
> -- executing async: localhost:21001
> select * from functional_parquet.alltypes where month < 3
> and id + random() < sleep(100);;
> -- 2021-08-26 05:46:42,208 INFO MainThread: Started query
> db4bd9b9ff7787ec:963015e700000000
> -- 2021-08-26 05:46:42,208 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:42,213 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: None
> -- 2021-08-26 05:46:42,213 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:43,214 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:43,218 INFO MainThread: Metric
> 'admission-controller.agg-num-running.default-pool' has reached desired
> value: 1
> -- executing async: localhost:21000
> select count(*) from functional.alltypes where month + random() < 3;
> -- 2021-08-26 05:46:43,673 INFO MainThread: Started query
> de46a56beb40a1c9:6f12b84c00000000
> -- 2021-08-26 05:46:43,673 INFO MainThread: Getting metric:
> admission-controller.local-num-queued.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:43,677 INFO MainThread: Metric
> 'admission-controller.local-num-queued.default-pool' has reached desired
> value: 1
> -- getting runtime profile operation:
> <tests.common.impala_connection.OperationHandle object at 0x7f3651f12710>
> -- closing query for operation handle:
> <tests.common.impala_connection.OperationHandle object at 0x7f3651f12710>
> -- closing query for operation handle:
> <tests.common.impala_connection.OperationHandle object at 0x7f3651f3ae50>
> -- 2021-08-26 05:46:43,680 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:43,684 INFO MainThread: Metric
> 'admission-controller.agg-num-running.default-pool' has reached desired
> value: 0
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
> SET
> client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords;
> SET mem_limit=4g;
> -- executing async: localhost:21001
> select * from functional_parquet.alltypes where month < 3
> and id + random() < sleep(100);;
> -- 2021-08-26 05:46:43,701 INFO MainThread: Started query
> 1841bd63d1384bff:7296726f00000000
> -- 2021-08-26 05:46:43,701 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:43,705 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:43,706 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:44,707 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:44,711 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:44,711 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:45,712 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:45,717 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:45,717 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:46,717 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:46,722 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:46,722 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:47,723 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:47,727 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:47,727 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:48,728 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:48,733 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:48,733 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:49,734 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:49,739 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:49,739 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:50,740 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:50,744 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:50,744 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:51,745 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:51,749 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:51,749 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:52,750 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:52,754 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:52,754 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:53,755 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:53,760 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:53,760 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:54,760 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:54,765 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:54,765 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:55,766 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:55,770 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:55,770 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:56,771 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:56,776 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:56,776 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:57,776 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:57,780 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:57,781 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:58,781 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:58,785 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:58,785 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:46:59,786 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:46:59,790 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:46:59,790 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:00,791 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:00,796 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:00,796 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:01,797 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:01,801 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:01,801 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:02,802 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:02,807 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:02,807 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:03,808 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:03,813 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:03,813 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:04,814 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:04,818 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:04,818 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:05,819 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:05,823 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:05,823 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:06,824 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:06,828 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:06,828 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:07,829 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:07,834 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:07,834 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:08,835 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:08,839 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:08,839 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:09,840 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:09,845 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:09,845 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:10,845 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:10,850 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:10,850 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:11,851 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:11,855 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:11,855 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:12,857 INFO MainThread: Getting metric:
> admission-controller.agg-num-running.default-pool from
> impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000
> -- 2021-08-26 05:47:12,861 INFO MainThread: Waiting for metric value
> 'admission-controller.agg-num-running.default-pool'=1. Current value: 0
> -- 2021-08-26 05:47:12,861 INFO MainThread: Sleeping 1s before next retry.
> -- 2021-08-26 05:47:13,862 INFO MainThread: Metric
> admission-controller.agg-num-running.default-pool did not reach value 1 in
> 30s. Failing...{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
