Wenzhe Zhou created IMPALA-10539: ------------------------------------ Summary: Creating tpcds testcase data Key: IMPALA-10539 URL: https://issues.apache.org/jira/browse/IMPALA-10539 Project: IMPALA Issue Type: Bug Components: Infrastructure Affects Versions: Impala 4.0 Environment: s3 build Reporter: Wenzhe Zhou
This issue happened a handful of times in the last few days on s3 build. Log messages: 22:28:09 Creating tpcds testcase data (logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)... 22:28:09 FAILED (Took: 0 min 3 sec) 22:28:12 '/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' failed. Tail of log: 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1653) 22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) 22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) 22:28:12 at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) 22:28:12 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) 22:28:12 at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197) 22:28:12 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 22:28:12 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 22:28:12 at java.lang.Thread.run(Thread.java:748) 22:28:12 Caused by: java.io.FileNotFoundException: copyFile(test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d, user/jenkins/.Trash/Current/test-warehouse/tpcds-testcase-data1614061691533/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d) on test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=:404 Not Found 22:28:12 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:260) 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3698) 22:28:12 ... 10 more 22:28:12 Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec= 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) 22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) 22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227) 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173) 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360) 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2027) 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375) 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2017) 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$18(S3AFileSystem.java:3700) 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115) 22:28:12 ... 11 more 22:28:12 rm: Failed to move to trash: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data: No such file or directory: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data. Consider using -skipTrash option 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system... 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped. 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete. 22:28:12 ++ report_build_error 39 22:28:12 +++ cd /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala 22:28:12 +++ awk 'NR == 39' /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh 22:28:12 ++ ERROR_MSG='hadoop fs -rm -r -f $TESTCASE_DATA_DIR' 22:28:12 +++ basename -- /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh 22:28:12 ++ FILENAME=create-tpcds-testcase-files.sh 22:28:12 ++ echo ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f '$TESTCASE_DATA_DIR' 22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR 22:28:12 ++ /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/bin/generate_junitxml.py --step create-tpcds-testcase-files --error 'Error in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR' 22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-tpcds-testcase-files.20210223_06_28_12.xml 22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-load-data.sh at line 48: 22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-load-data.20210223_06_28_12.xml 22:28:12 + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.' 22:28:12 buildall.sh -format -snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed. 22:28:12 + RET_VAL=1 22:28:12 + [[ DEBUG = \A\S\A\N ]] 22:28:12 + [[ DEBUG = \S\T\R\E\S\S ]] 22:28:12 + [[ DEBUG = \T\S\A\N ]] 22:28:12 + '[' 1 -ne 0 ']' 22:28:12 + echo 22:28:12 22:28:12 + echo 22:28:12 22:28:12 + echo '**** BUILD FAILED! ****' 22:28:12 **** BUILD FAILED! **** -- This message was sent by Atlassian Jira (v8.3.4#803005)