[ https://issues.apache.org/jira/browse/IMPALA-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17288919#comment-17288919 ]
Wenzhe Zhou commented on IMPALA-10539: -------------------------------------- In some cases, it failed when running bin/run-all-tests.sh in s3 builds with following error messages. *04:48:27* ERROR in /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/run-all-tests.sh at line 198: hdfs dfs -ls -R /test-warehouse > ${IMPALA_LOGS_DIR}/file-list-begin-${i}.log 2>&1*04:52:12* Generated: /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.run-all-tests.20210219_12_52_12.xml*04:52:12* + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-asf-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-1284-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-asf-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-1284.txt failed.'*04:52:12* buildall.sh -format -snapshot_file /data/jenkins/workspace/impala-asf-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-1284-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-asf-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-1284.txt failed. > Creating tpcds testcase data failed in s3 build > ----------------------------------------------- > > Key: IMPALA-10539 > URL: https://issues.apache.org/jira/browse/IMPALA-10539 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Affects Versions: Impala 4.0 > Environment: s3 build > Reporter: Wenzhe Zhou > Priority: Major > Labels: broken-build > > This issue happened a handful of times in the last few days for different s3 > builds. > Log messages: > 22:28:09 Creating tpcds testcase data (logging to > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)... > 22:28:09 FAILED (Took: 0 min 3 sec) > 22:28:12 > '/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' > failed. Tail of log: > 22:28:12 at > org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1653) > 22:28:12 at > org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636) > 22:28:12 at > org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600) > 22:28:12 at > org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62) > 22:28:12 at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) > 22:28:12 at > org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197) > 22:28:12 at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 22:28:12 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 22:28:12 at java.lang.Thread.run(Thread.java:748) > 22:28:12 Caused by: java.io.FileNotFoundException: > copyFile(test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d, > > user/jenkins/.Trash/Current/test-warehouse/tpcds-testcase-data1614061691533/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d) > on > test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d: > com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: > Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: > DA4BB61317C3469E; S3 Extended Request ID: > oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; > Proxy: null), S3 Extended Request ID: > oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=:404 > Not Found > 22:28:12 at > org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:260) > 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117) > 22:28:12 at > org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3698) > 22:28:12 ... 10 more > 22:28:12 Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not > Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; > Request ID: DA4BB61317C3469E; S3 Extended Request ID: > oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; > Proxy: null), S3 Extended Request ID: > oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec= > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) > 22:28:12 at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) > 22:28:12 at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227) > 22:28:12 at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173) > 22:28:12 at > com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360) > 22:28:12 at > org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2027) > 22:28:12 at > org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412) > 22:28:12 at > org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375) > 22:28:12 at > org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2017) > 22:28:12 at > org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$18(S3AFileSystem.java:3700) > 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115) > 22:28:12 ... 11 more > 22:28:12 rm: Failed to move to trash: > s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data: No such file > or directory: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data. > Consider using -skipTrash option > 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: Stopping > s3a-file-system metrics system... > 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system > metrics system stopped. > 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system > metrics system shutdown complete. > 22:28:12 ++ report_build_error 39 > 22:28:12 +++ cd > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala > 22:28:12 +++ awk 'NR == 39' > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh > 22:28:12 ++ ERROR_MSG='hadoop fs -rm -r -f $TESTCASE_DATA_DIR' > 22:28:12 +++ basename – > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh > 22:28:12 ++ FILENAME=create-tpcds-testcase-files.sh > 22:28:12 ++ echo ERROR in > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh > at line 39: hadoop fs -rm -r -f '$TESTCASE_DATA_DIR' > 22:28:12 ERROR in > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh > at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR > 22:28:12 ++ > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/bin/generate_junitxml.py > --step create-tpcds-testcase-files --error 'Error in > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh > at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR' > 22:28:12 Generated: > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-tpcds-testcase-files.20210223_06_28_12.xml > 22:28:12 ERROR in > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-load-data.sh > at line 48: > 22:28:12 Generated: > /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-load-data.20210223_06_28_12.xml > 22:28:12 + echo 'buildall.sh ' -format '-snapshot_file > /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz' > '-metastore_snapshot_file > /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt > failed.' > 22:28:12 buildall.sh -format -snapshot_file > /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz > -metastore_snapshot_file > /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt > failed. > 22:28:12 + RET_VAL=1 > 22:28:12 + [[ DEBUG = \A\S\A\N ]] > 22:28:12 + [[ DEBUG = \S\T\R\E\S\S ]] > 22:28:12 + [[ DEBUG = \T\S\A\N ]] > 22:28:12 + '[' 1 -ne 0 ']' > 22:28:12 + echo > 22:28:12 > 22:28:12 + echo > 22:28:12 > 22:28:12 + echo '**** BUILD FAILED! ****' > 22:28:12 **** BUILD FAILED! **** -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org