[
https://issues.apache.org/jira/browse/IMPALA-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282208#comment-17282208
]
Riza Suminto commented on IMPALA-10497:
---------------------------------------
I'm able to reproduce this test failure after several retry. Still not sure
what is the root cause of this flakiness though.
I'll try to increase the sleep time and see if the flakiness is gone.
> test_no_fd_caching_on_cached_data failing
> -----------------------------------------
>
> Key: IMPALA-10497
> URL: https://issues.apache.org/jira/browse/IMPALA-10497
> Project: IMPALA
> Issue Type: Bug
> Reporter: Bikramjeet Vig
> Assignee: Riza Suminto
> Priority: Major
> Labels: broken-build
>
> {noformat}
> Error Message
> assert 1 == 0 + where 1 = <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>>() +
> where <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>> =
> <test_hdfs_fd_caching.TestHdfsFdCaching object at
> 0x7f22dfe5aa10>.cached_handles
> Stacktrace
> custom_cluster/test_hdfs_fd_caching.py:202: in
> test_no_fd_caching_on_cached_data
> assert self.cached_handles() == 0
> E assert 1 == 0
> E + where 1 = <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>>()
> E + where <bound method TestHdfsFdCaching.cached_handles of
> <test_hdfs_fd_caching.TestHdfsFdCaching object at 0x7f22dfe5aa10>> =
> <test_hdfs_fd_caching.TestHdfsFdCaching object at
> 0x7f22dfe5aa10>.cached_handles
> Standard Error
> -- 2021-02-08 06:40:41,413 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--max_cached_file_handles=16
> --unused_file_handle_timeout_sec=5 --data_cache=/tmp:500MB
> --always_use_data_cache=true ' '--state_store_args=None '
> '--catalogd_args=--load_catalog_in_background=false '
> --impalad_args=--default_query_options=
> 06:40:42 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 06:40:42 MainThread: Starting State Store logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 06:40:42 MainThread: Starting Catalog Service logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 06:40:42 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 06:40:42 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 06:40:42 MainThread: Starting Impala Daemon logging to
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 06:40:45 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:45 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:45 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> 06:40:45 MainThread: Debug webpage not yet available: ('Connection aborted.',
> error(111, 'Connection refused'))
> 06:40:47 MainThread: Debug webpage did not become available in expected time.
> 06:40:47 MainThread: Waiting for num_known_live_backends=3. Current value:
> None
> 06:40:48 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:48 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> 06:40:48 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 06:40:49 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:49 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> 06:40:49 MainThread: num_known_live_backends has reached value: 3
> 06:40:49 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:49 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25001
> 06:40:49 MainThread: num_known_live_backends has reached value: 3
> 06:40:50 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 06:40:50 MainThread: Getting num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25002
> 06:40:50 MainThread: num_known_live_backends has reached value: 3
> 06:40:50 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> -- 2021-02-08 06:40:51,049 DEBUG MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> -- 2021-02-08 06:40:51,049 INFO MainThread: Getting metric:
> statestore.live-backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25010
> -- 2021-02-08 06:40:51,050 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,052 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4
> -- 2021-02-08 06:40:51,052 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25000
> -- 2021-02-08 06:40:51,053 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,054 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2021-02-08 06:40:51,054 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25001
> -- 2021-02-08 06:40:51,055 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,056 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2021-02-08 06:40:51,056 DEBUG MainThread: Getting
> num_known_live_backends from
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:25002
> -- 2021-02-08 06:40:51,057 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:51,058 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-02-08 06:40:51,307 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-02-08 06:40:51,332 INFO MainThread: Closing active operation
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- connecting to:
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> drop database if exists cachefd cascade;
> -- 2021-02-08 06:40:51,527 INFO MainThread: Started query
> c94d37c9548851e5:7c7fb33300000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> create database cachefd;
> -- 2021-02-08 06:40:51,577 INFO MainThread: Started query
> f445a2e444e61b11:246e8dfd00000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> create table cachefd.simple(id int, col1 int, col2 int) stored as parquet;
> -- 2021-02-08 06:40:51,706 INFO MainThread: Started query
> 504e5f830c41c2ba:efc4cdad00000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> insert into cachefd.simple values(0,0,0), (1,1,1), (2,2,2), (3,3,3), (4,4,4),
> (5,5,5), (6,6,6), (7,7,7), (8,8,8), (9,9,9), (10,10,10), (11,11,11),
> (12,12,12), (13,13,13), (14,14,14), (15,15,15), (16,16,16), (17,17,17),
> (18,18,18), (19,19,19), (20,20,20), (21,21,21), (22,22,22), (23,23,23),
> (24,24,24), (25,25,25), (26,26,26), (27,27,27), (28,28,28), (29,29,29),
> (30,30,30), (31,31,31), (32,32,32), (33,33,33), (34,34,34), (35,35,35),
> (36,36,36), (37,37,37), (38,38,38), (39,39,39), (40,40,40), (41,41,41),
> (42,42,42), (43,43,43), (44,44,44), (45,45,45), (46,46,46), (47,47,47),
> (48,48,48), (49,49,49), (50,50,50), (51,51,51), (52,52,52), (53,53,53),
> (54,54,54), (55,55,55), (56,56,56), (57,57,57), (58,58,58), (59,59,59),
> (60,60,60), (61,61,61), (62,62,62), (63,63,63), (64,64,64), (65,65,65),
> (66,66,66), (67,67,67), (68,68,68), (69,69,69), (70,70,70), (71,71,71),
> (72,72,72), (73,73,73), (74,74,74), (75,75,75), (76,76,76), (77,77,77),
> (78,78,78), (79,79,79), (80,80,80), (81,81,81), (82,82,82), (83,83,83),
> (84,84,84), (85,85,85), (86,86,86), (87,87,87), (88,88,88), (89,89,89),
> (90,90,90), (91,91,91), (92,92,92), (93,93,93), (94,94,94), (95,95,95),
> (96,96,96), (97,97,97), (98,98,98), (99,99,99);
> -- 2021-02-08 06:40:52,482 INFO MainThread: Started query
> 7041325f1c73e372:2ec9e2ef00000000
> -- 2021-02-08 06:40:52,742 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,745 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,749 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,754 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,758 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:52,761 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> use functional;
> -- 2021-02-08 06:40:52,772 INFO MainThread: Started query
> 194c55cdb4433725:bff1f94d00000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> select * from cachefd.simple;
> -- 2021-02-08 06:40:52,842 INFO MainThread: Started query
> 3a47160da328b7dd:983c9ca700000000
> -- 2021-02-08 06:40:53,053 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,057 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,061 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,064 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,068 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,071 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,074 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,077 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:40:53,081 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,094 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,107 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,120 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> SET
> client_identifier=custom_cluster/test_hdfs_fd_caching.py::TestHdfsFdCaching::()::test_no_fd_caching_on_cached_data[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> use functional;
> -- 2021-02-08 06:41:03,137 INFO MainThread: Started query
> 2a4e414134944c87:f07a223200000000
> -- executing against
> impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com:21000
> select * from cachefd.simple;
> -- 2021-02-08 06:41:03,146 INFO MainThread: Started query
> 7a4a8d4994fe200d:ec8c84ee00000000
> -- 2021-02-08 06:41:03,256 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,261 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> -- 2021-02-08 06:41:03,265 INFO MainThread: Starting new HTTP connection
> (1): impala-ec2-centos74-r5-4xlarge-ondemand-02df.vpc.cloudera.com
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
