[
https://issues.apache.org/jira/browse/HADOOP-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619073#comment-17619073
]
Jimmy Wong commented on HADOOP-18233:
-------------------------------------
Integration test results for s3a & s3n @ us-west-2:
# mvn clean verify
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default) @
hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[WARNING] Tests run: 16, Failures: 0, Errors: 0, Skipped: 4, Time elapsed:
19.714 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.884
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[WARNING] Tests run: 72, Failures: 0, Errors: 0, Skipped: 24, Time elapsed:
82.936 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.984 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer
[INFO] Running
org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 17, Time elapsed:
5.457 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.253 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
20.828 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[WARNING] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 82.4
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.82 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.1 s -
in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.091 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractContentSummary
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.374 s
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractContentSummary
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.391
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 2.924
s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
[INFO] Running org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.315 s
- in org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.967
s - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.89 s
<<< FAILURE! - in
org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[ERROR]
testBytesReadWithStream(org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic)
Time elapsed: 0.89 s <<< FAILURE!
java.lang.AssertionError: Mismatch in number of FS bytes read by InputStreams
expected:<2048> but was:<19599544>
at
org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic.testBytesReadWithStream(ITestS3AFileSystemStatistic.java:72)
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestAggregateIOStatistics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.388 s
- in org.apache.hadoop.fs.s3a.statistics.ITestAggregateIOStatistics
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestAWSStatisticCollection
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.783 s
- in org.apache.hadoop.fs.s3a.statistics.ITestAWSStatisticCollection
[INFO] Running
org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.139 s
- in org.apache.hadoop.fs.s3a.statistics.ITestS3AContractStreamIOStatistics
[INFO] Running
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.935
s - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol
[INFO] Running
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.619
s - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol
[INFO] Running
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[WARNING] Tests run: 22, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
55.011 s - in
org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.092
s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol
[INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
3.967 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits
[INFO] Running
org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.497
s - in org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.522 s
- in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.79 s
- in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.898 s
- in org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 14, Time elapsed:
6.91 s - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestRenameDeleteRace
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.164 s
- in org.apache.hadoop.fs.s3a.impl.ITestRenameDeleteRace
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[WARNING] Tests run: 26, Failures: 0, Errors: 0, Skipped: 26, Time elapsed:
10.285 s - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes
[INFO] Running org.apache.hadoop.fs.s3a.impl.ITestXAttrCost
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.89 s -
in org.apache.hadoop.fs.s3a.impl.ITestXAttrCost
[INFO] Running
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.579 s
- in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
50.233 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 s -
in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.45 s -
in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Running
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.45 s
- in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[INFO] Running
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[WARNING] Tests run: 73, Failures: 0, Errors: 0, Skipped: 4, Time elapsed:
154.613 s - in
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.005 s
- in org.apache.hadoop.fs.s3a.ITestS3ABucketExistence
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.851 s
- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.207 s
- in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Running org.apache.hadoop.fs.s3a.ITestDowngradeSyncable
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.49 s -
in org.apache.hadoop.fs.s3a.ITestDowngradeSyncable
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerToolRootOperations
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.317 s
- in org.apache.hadoop.fs.s3a.tools.ITestMarkerToolRootOperations
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.017
s - in org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.789 s
- in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
3.845 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.378
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AMultipartUploadSizeLimits
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.066
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AMultipartUploadSizeLimits
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
3.966 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
0.153 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
4.056 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 2.269
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.386
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 3.107
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.717
s - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
3.83 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.378
s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
3.927 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContentEncoding
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.467 s
- in org.apache.hadoop.fs.s3a.ITestS3AContentEncoding
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.037 s
- in org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.677 s
- in org.apache.hadoop.fs.s3a.ITestS3AUnbuffer
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestJceksIO
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.558 s
- in org.apache.hadoop.fs.s3a.auth.ITestJceksIO
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestCustomSigner
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.307 s
- in org.apache.hadoop.fs.s3a.auth.ITestCustomSigner
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.375
s - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[WARNING] Tests run: 25, Failures: 0, Errors: 0, Skipped: 25, Time elapsed:
9.623 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.012 s
- in org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.803
s - in
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed:
5.591 s - in
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
[INFO] Running
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 2.794
s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
13.771 s - in org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
[INFO] Running org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed:
6.718 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumedRoleCommitOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.457 s
- in org.apache.hadoop.fs.s3a.ITestS3AMetrics
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.158 s
- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AStorageClass
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.159
s - in org.apache.hadoop.fs.s3a.ITestS3AStorageClass
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.059 s
- in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.187 s
- in org.apache.hadoop.fs.s3a.ITestS3AClosedFS
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.705
s - in org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ADeleteOnExit
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.259 s
- in org.apache.hadoop.fs.s3a.ITestS3ADeleteOnExit
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.582 s
- in org.apache.hadoop.fs.s3a.ITestS3AMultipartUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.278
s - in org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.281
s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.727 s
<<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[ERROR] testCreatedObjectsHaveACLs(org.apache.hadoop.fs.s3a.ITestS3ACannedACLs)
Time elapsed: 0.726 s <<< ERROR!
org.apache.hadoop.fs.s3a.audit.AuditFailureException:
7b1df527-cb0f-42ce-aa2c-cf107864a1e1-00011029 unaudited operation executing a
request outside an audit span
\{com.amazonaws.services.s3.model.GetObjectAclRequest size=0, mutating=true}
at
org.apache.hadoop.fs.s3a.ITestS3ACannedACLs.assertObjectHasLoggingGrant(ITestS3ACannedACLs.java:97)
at
org.apache.hadoop.fs.s3a.ITestS3ACannedACLs.testCreatedObjectsHaveACLs(ITestS3ACannedACLs.java:71)
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.794 s
- in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.049 s
- in org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.724 s
- in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.183 s
- in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.409 s
- in org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.193
s - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.302 s
- in org.apache.hadoop.fs.s3a.ITestS3APrefetchingInputStream
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditAccessChecks
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.654 s
- in org.apache.hadoop.fs.s3a.audit.ITestAuditAccessChecks
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditManager
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.76 s -
in org.apache.hadoop.fs.s3a.audit.ITestAuditManager
[INFO] Running org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.743 s
- in org.apache.hadoop.fs.s3a.audit.ITestAuditManagerDisabled
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.129 s
- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.045
s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.86 s -
in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.465 s
- in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 3, Time elapsed:
57.969 s - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.851
s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardTool
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in
org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.336 s
- in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.783
s - in org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.927 s
- in org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 9.116
s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.07 s
- in org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI
[INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.514
s - in org.apache.hadoop.fs.s3a.select.ITestS3Select
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.445 s
- in org.apache.hadoop.fs.s3a.performance.ITestS3AOpenCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AMkdirCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.516 s
- in org.apache.hadoop.fs.s3a.performance.ITestS3AMkdirCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ARenameCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.649 s
- in org.apache.hadoop.fs.s3a.performance.ITestS3ARenameCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3AMiscOperationCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.97 s -
in org.apache.hadoop.fs.s3a.performance.ITestS3AMiscOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.615
s - in org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestUnbufferDraining
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.489 s
- in org.apache.hadoop.fs.s3a.performance.ITestUnbufferDraining
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestCreateFileCost
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.71 s
- in org.apache.hadoop.fs.s3a.performance.ITestCreateFileCost
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.431 s
- in org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.665 s
- in org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.307
s - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.834 s
- in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -
in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 2.751
s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.566 s
- in org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestS3AFileSystemStatistic.testBytesReadWithStream:72->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
Mismatch in number of FS bytes read by InputStreams expected:<2048> but
was:<19599544>
[ERROR] Errors:
[ERROR]
ITestS3ACannedACLs.testCreatedObjectsHaveACLs:71->assertObjectHasLoggingGrant:97
» AuditFailure
[INFO]
[ERROR] Tests run: 1277, Failures: 1, Errors: 1, Skipped: 266
[INFO]
[INFO]
# mvn -Dtest=skip
-Dit.test=ITestS3AFileSystemStatistic#testBytesReadWithStream,ITestS3ACannedACLs#testCreatedObjectsHaveACLs
clean verify
[INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default) @
hadoop-aws ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.797 s
- in org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.692 s
- in org.apache.hadoop.fs.s3a.ITestS3ACannedACLs
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
> Possible race condition with TemporaryAWSCredentialsProvider
> ------------------------------------------------------------
>
> Key: HADOOP-18233
> URL: https://issues.apache.org/jira/browse/HADOOP-18233
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth, fs/s3
> Affects Versions: 3.3.1
> Environment: spark v3.2.0
> hadoop-aws v3.3.1
> java version 1.8.0_265 via zulu-8
> Reporter: Jason Sleight
> Priority: Major
> Labels: pull-request-available
>
> I'm in the process of upgrading spark+hadoop versions for my workflows and
> observing a weird behavior regression. I'm setting
> {code:java}
> spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
> spark.sql.catalogImplementation=hive
> spark.hadoop.aws.region=us-west-2
> ...many other things, I think these might be the relevant ones though...{code}
> in Spark config and I'm observing some non-fatal warnings/exceptions (see
> below for some examples). The warnings/exceptions randomly appear for some
> tasks, which causes them to fail, but then when Spark retries the task it
> will succeed. The initial tasks don't always fail either, just sometimes.
> I also found that if I switch to a SimpleAWSCredentials and use static keys,
> then I don't see any issues.
> My old setup was spark v3.0.2 with hadoop-aws v3.2.1 which also does not have
> these warnings/exceptions.
> From reading some other tickets I thought perhaps adding
> {code:java}
> spark.sql.hive.metastore.sharedPrefixes=com.amazonaws {code}
> would help, but it did not.
> Appreciate any suggestions for how to proceed or debug further :)
>
> Example stack traces:
> First one for an s3 read
> {code:java}
> WARN TaskSetManager: Lost task 27.0 in stage 4.0 (TID 29) (<ip> executor
> 13): java.nio.file.AccessDeniedException:
> s3a://bucket/path/to/part.snappy.parquet:
> org.apache.hadoop.fs.s3a.CredentialInitializationException: Provider
> TemporaryAWSCredentialsProvider has no credentials
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:206)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3289)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
> at
> org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:268)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$1(ParquetFileFormat.scala:267)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(ParquetFileFormat.scala:270)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:164)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
> at
> org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:522)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.columnartorow_nextBatch_0$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
> at
> org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
> at org.apache.spark.scheduler.Task.run(Task.scala:131)
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.s3a.CredentialInitializationException:
> Provider TemporaryAWSCredentialsProvider has no credentials
> at
> org.apache.hadoop.fs.s3a.auth.AbstractSessionCredentialsProvider.getCredentials(AbstractSessionCredentialsProvider.java:130)
> at
> org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
> at
> com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6408)
> at
> com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6381)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5422)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1367)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2066)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2056)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2032)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3273)
> ... 27 more{code}
> And here is one for an s3 write which is similar but slightly different
> {code:java}
> WARN TaskSetManager: Lost task 21.0 in stage 78.0 (TID 1358) (<ip> executor
> 11): org.apache.spark.SparkException: Task failed while writing rows.
> at
> org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:500)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:321)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$16(FileFormatWriter.scala:229)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:131)
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.AccessDeniedException:
> s3a://bucket/path/to/_temporary/0/_temporary/attempt_0123456789:
> org.apache.hadoop.fs.s3a.CredentialInitializationException: Provider
> TemporaryAWSCredentialsProvider has no credentials
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:206)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3289)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4263)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:674)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:663)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:61)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:269)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$commit$1(FileFormatDataWriter.scala:107)
> at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:605)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:107)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:305)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:311)
> ... 9 more
> Caused by: org.apache.hadoop.fs.s3a.CredentialInitializationException:
> Provider TemporaryAWSCredentialsProvider has no credentials
> at
> org.apache.hadoop.fs.s3a.auth.AbstractSessionCredentialsProvider.getCredentials(AbstractSessionCredentialsProvider.java:130)
> at
> org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1266)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:842)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:792)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
> at
> com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6408)
> at
> com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6381)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5422)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1367)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2066)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2056)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2032)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3273)
> ... 23 more {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
