[
https://issues.apache.org/jira/browse/HADOOP-11488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14284046#comment-14284046
]
Thomas Demoor commented on HADOOP-11488:
----------------------------------------
[[email protected]] Tests pass against bucket in eu-west1 (Ireland).
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 186.434 sec -
in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 366.926 sec -
in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.551 sec - in
org.apache.hadoop.fs.s3a.TestS3AConfiguration
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.301 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.643 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.497 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.127 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.894 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 9.769 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.816 sec -
in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.431 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.882 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.157 sec -
in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 7.307 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.792 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.093 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.693 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Running org.apache.hadoop.fs.s3.TestS3Credentials
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.13 sec - in
org.apache.hadoop.fs.s3.TestS3Credentials
Running org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.427 sec - in
org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.068 sec - in
org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Running org.apache.hadoop.fs.s3.TestINode
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.145 sec - in
org.apache.hadoop.fs.s3.TestINode
Running org.apache.hadoop.fs.s3.TestS3FileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.475 sec - in
org.apache.hadoop.fs.s3.TestS3FileSystem
Running org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.004 sec - in
org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.512 sec - in
org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.634 sec - in
org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
Results :
Tests run: 247, Failures: 0, Errors: 0, Skipped: 6
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:31.091s
[INFO] Finished at: Tue Jan 20 17:30:50 CET 2015
[INFO] Final Memory: 12M/327M
[INFO] ------------------------------------------------------------------------
{code}
[~daisuke.kobayashi] your test durations are quite long (compared to mine, see
above) so I assume you're connection is too slow to meaningfully run the tests
(do you run against a bucket in a region nearby?). You encounter the test
timeout before possibly hitting the (elongated by this patch) connection
timeout.
As this issue looks like it has some people fairly new to testing S3A, a small
side note: I will extend the testing guidelines in the documentation while I
pass by to document my recently accepted patches. A quick preview:
- When testing against Amazon S3, tests can sometimes randomly fail due to
eventual consistency. I frequently get this on
TestS3AFileSystemContract.testRenameRootDirForbidden. Retry and it should run
fine (or use a strong consistent object store :p)
- Beware test failures during multi-part uploads. Completed parts of a failed
multi-part upload do not show up in "ls" but linger (and are billed) eternally.
One can remove them manually or through purging: setting fs.s3a.multipart.purge
to true.
> Difference in default connection timeout for S3A FS
> ---------------------------------------------------
>
> Key: HADOOP-11488
> URL: https://issues.apache.org/jira/browse/HADOOP-11488
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Harsh J
> Assignee: Daisuke Kobayashi
> Priority: Minor
> Attachments: HADOOP-11488.patch, HADOOP-11488.patch
>
>
> The core-default.xml defines fs.s3a.connection.timeout as 5000, and the code
> under hadoop-tools/hadoop-aws defines it as 50000.
> We should update the former to 50s so it gets taken properly, as we're also
> noticing that 5s is often too low, especially in cases such as large DistCp
> operations (which fail with {{Read timed out}} errors from the S3 service).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
