[ https://issues.apache.org/jira/browse/IMPALA-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-7122: ---------------------------------- Attachment: load-functional-query.log > Data load failure: Failed to replace a bad datanode on the existing pipeline > due to no more good datanodes being available to try > --------------------------------------------------------------------------------------------------------------------------------- > > Key: IMPALA-7122 > URL: https://issues.apache.org/jira/browse/IMPALA-7122 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Affects Versions: Impala 3.1.0 > Reporter: Tim Armstrong > Assignee: Joe McDonnell > Priority: Critical > Labels: flaky > Attachments: > impalad.ec2-m2-4xlarge-centos-6-4-0570.vpc.cloudera.com.jenkins.log.INFO.20180604-205755.5587, > load-functional-query.log > > > {noformat} > 20:58:29 Started Loading functional-query data in background; pid 6813. > 20:58:29 Loading functional-query data (logging to > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-functional-query.log)... > > 20:58:29 Started Loading TPC-H data in background; pid 6814. > 20:58:29 Loading TPC-H data (logging to > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-tpch.log)... > > 20:58:29 Started Loading TPC-DS data in background; pid 6815. > 20:58:29 Loading TPC-DS data (logging to > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-tpcds.log)... > > 21:35:26 FAILED (Took: 36 min 57 sec) > 21:35:26 'load-data functional-query exhaustive' failed. Tail of log: > 21:35:26 at > org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) > 21:35:26 18/06/04 21:20:29 WARN hdfs.DataStreamer: Error Recovery for > BP-1407206351-127.0.0.1-1528170335185:blk_1073743620_2799 in pipeline > [DatanodeInfoWithStorage[127.0.0.1:31000,DS-37cfc57c-ab39-443c-80c9-e440cb18b63d,DISK], > > DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK], > > DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]]: > datanode > 0(DatanodeInfoWithStorage[127.0.0.1:31000,DS-37cfc57c-ab39-443c-80c9-e440cb18b63d,DISK]) > is bad. > 21:35:26 18/06/04 21:21:29 INFO hdfs.DataStreamer: Exception in > createBlockOutputStream blk_1073743620_2799 > 21:35:26 java.io.IOException: Got error, status=ERROR, status message , ack > with firstBadLink as 127.0.0.1:31002 > 21:35:26 at > org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134) > 21:35:26 at > org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) > 21:35:26 18/06/04 21:21:29 WARN hdfs.DataStreamer: Error Recovery for > BP-1407206351-127.0.0.1-1528170335185:blk_1073743620_2799 in pipeline > [DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK], > > DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]]: > datanode > 1(DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]) > is bad. > 21:35:26 18/06/04 21:21:29 WARN hdfs.DataStreamer: DataStreamer Exception > 21:35:26 java.io.IOException: Failed to replace a bad datanode on the > existing pipeline due to no more good datanodes being available to try. > (Nodes: > current=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]], > > original=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]]). > The current failed datanode replacement policy is DEFAULT, and a client may > configure this via > 'dfs.client.block.write.replace-datanode-on-failure.policy' in its > configuration. > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) > 21:35:26 at > org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) > 21:35:26 put: Failed to replace a bad datanode on the existing pipeline due > to no more good datanodes being available to try. (Nodes: > current=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]], > > original=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]]). > The current failed datanode replacement policy is DEFAULT, and a client may > configure this via > 'dfs.client.block.write.replace-datanode-on-failure.policy' in its > configuration. > 21:35:26 18/06/04 21:24:25 INFO hdfs.DFSClient: Could not complete > /test-warehouse/testescape_17_crlf/126._COPYING_ retrying... > 21:35:26 be loaded. > 21:35:26 Empty base table load for chars_tiny. Skipping load generation > 21:35:26 HDFS path: /test-warehouse/widetable_250_cols does not exists or is > empty. Data will be loaded. > 21:35:26 HDFS path: /test-warehouse/widetable_500_cols does not exists or is > empty. Data will be loaded. > 21:35:26 HDFS path: /test-warehouse/widetable_1000_cols does not exists or is > empty. Data will be loaded. > 21:35:26 Skipping 'functional.avro_decimal_tbl' due to include constraint > match. > 21:35:26 Skipping 'functional.no_avro_schema' due to include constraint match. > 21:35:26 HDFS path: /test-warehouse/table_no_newline does not exists or is > empty. Data will be loaded. > 21:35:26 Empty base table load for table_no_newline. Skipping load generation > 21:35:26 HDFS path: /test-warehouse/table_no_newline_part does not exists or > is empty. Data will be loaded. > 21:35:26 Empty base table load for table_no_newline_part. Skipping load > generation > 21:35:26 HDFS path: /test-warehouse/testescape_16_lf does not exists or is > empty. Data will be loaded. > 21:35:26 Empty base table load for testescape_16_lf. Skipping load generation > 21:35:26 HDFS path: /test-warehouse/testescape_16_crlf does not exists or is > empty. Data will be loaded. > 21:35:26 Empty base table load for testescape_16_crlf. Skipping load > generation > 21:35:26 HDFS path: /test-warehouse/testescape_17_lf does not exists or is > empty. Data will be loaded. > 21:35:26 Empty base table load for testescape_17_lf. Skipping load generation > 21:35:26 Traceback (most recent call last): > 21:35:26 File > "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", > line 836, in <module> > 21:35:26 test_vectors, sections, include_constraints, > exclude_constraints, only_constraints) > 21:35:26 File > "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", > line 595, in generate_statements > 21:35:26 load = eval_section(section['LOAD']) > 21:35:26 File > "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", > line 533, in eval_section > 21:35:26 assert p.returncode == 0 > 21:35:26 AssertionError > 21:35:26 21:35:26 Error generating schema statements for workload: > functional-query > 21:35:26 Background task Loading functional-query data (pid 6813) failed. > 21:48:12 FAILED (Took: 49 min 43 sec) > 21:48:12 'load-data tpch core' failed. Tail of log: > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-none-none.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-gzip-block.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-snap-block.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-none-none.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-snap-block.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-parquet-none-none.sql > 21:48:13 20:59:36 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-rc-none-none.sql > 21:48:13 20:59:53 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-parquet-none-none.sql > 21:48:13 20:59:53 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-orc-def-block.sql > 21:48:13 20:59:53 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-none-none.sql > 21:48:13 20:59:53 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-kudu-none-none.sql > 21:48:13 20:59:53 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-gzip-block.sql > 21:48:13 20:59:53 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql > 21:48:13 20:59:53 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-snap-block.sql > 21:48:13 20:59:54 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-none-none.sql > 21:48:13 20:59:59 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-snap-block.sql > 21:48:13 20:59:59 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-rc-none-none.sql > 21:48:13 20:59:59 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-orc-def-block.sql > 21:48:13 21:00:21 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-kudu-none-none.sql > 21:48:13 21:00:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-none-none.sql > 21:48:13 21:01:21 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-none-none.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-gzip-block.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-gzip-block.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-rc-none-none.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-none-none.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-snap-block.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-snap-block.sql > 21:48:13 21:01:21 Beginning execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-orc-def-block.sql > 21:48:13 21:21:55 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-snap-block.sql > 21:48:13 21:22:22 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-orc-def-block.sql > 21:48:13 21:26:17 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-gzip-block.sql > 21:48:13 21:28:15 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-rc-none-none.sql > 21:48:13 21:29:13 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-gzip-block.sql > 21:48:13 21:29:43 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-snap-block.sql > 21:48:13 21:37:08 Finished execution of hive SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-none-none.sql > 21:48:13 21:37:08 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/invalidate-tpch-core-impala-generated.sql > 21:48:13 21:37:31 Finished execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/invalidate-tpch-core-impala-generated.sql > 21:48:13 21:37:31 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-kudu-none-none.sql > 21:48:13 21:37:31 Beginning execution of impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql > 21:48:13 21:48:12 Error executing impala SQL: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql > See: > /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql.log > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org