[
https://issues.apache.org/jira/browse/HADOOP-11488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283614#comment-14283614
]
Daisuke Kobayashi commented on HADOOP-11488:
--------------------------------------------
[[email protected]], I've ran the test for hadoop-aws locally and here's the
results. Please review and let me know if anything is needed from my side.
Thanks!
{noformat}
daisuke:hadoop-aws/ (HADOOP-11488✗) $ mvn test -Ptests-on
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Amazon Web Services support 3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-aws ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.2:resources (default-resources) @
hadoop-aws ---
[INFO] Using default encoding to copy filtered resources.
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-aws
---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @
hadoop-aws ---
[INFO] Using default encoding to copy filtered resources.
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @
hadoop-aws ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-aws ---
[INFO] Surefire report directory:
/src/apache/hadoop/hadoop-tools/hadoop-aws/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 53.492 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.338 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.846 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.969 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.803 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.372 sec - in
org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.805 sec -
in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 37.581 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.566 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.025 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.374 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.436 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.648 sec - in
org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.111 sec -
in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.918 sec - in
org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Running org.apache.hadoop.fs.s3.TestINode
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in
org.apache.hadoop.fs.s3.TestINode
Running org.apache.hadoop.fs.s3.TestS3Credentials
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec - in
org.apache.hadoop.fs.s3.TestS3Credentials
Running org.apache.hadoop.fs.s3.TestS3FileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.441 sec - in
org.apache.hadoop.fs.s3.TestS3FileSystem
Running org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.27 sec - in
org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,904.4 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
testBulkRenameAndDelete(org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles)
Time elapsed: 1,800.007 sec <<< ERROR!
java.lang.Exception: test timed out after 1800000 milliseconds
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at
org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at
com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
at
com.amazonaws.services.s3.transfer.TransferManager.copy(TransferManager.java:1285)
at
com.amazonaws.services.s3.transfer.TransferManager.copy(TransferManager.java:1223)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:1085)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:542)
at
org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles.testBulkRenameAndDelete(TestS3ADeleteManyFiles.java:95)
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.635 sec - in
org.apache.hadoop.fs.s3a.TestS3AConfiguration
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,010.135 sec
- in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.543 sec - in
org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
Running org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 757.153 sec -
in org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.753 sec - in
org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
Results :
Tests in error:
TestS3ADeleteManyFiles.testBulkRenameAndDelete:95 ? test timed out after
1800...
Tests run: 247, Failures: 0, Errors: 1, Skipped: 6
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:15:46.776s
[INFO] Finished at: Tue Jan 20 17:45:28 JST 2015
[INFO] Final Memory: 10M/310M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on
project hadoop-aws: There are test failures.
[ERROR]
[ERROR] Please refer to
/src/apache/hadoop/hadoop-tools/hadoop-aws/target/surefire-reports for the
individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
{noformat}
> Difference in default connection timeout for S3A FS
> ---------------------------------------------------
>
> Key: HADOOP-11488
> URL: https://issues.apache.org/jira/browse/HADOOP-11488
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Harsh J
> Assignee: Daisuke Kobayashi
> Priority: Minor
> Attachments: HADOOP-11488.patch, HADOOP-11488.patch
>
>
> The core-default.xml defines fs.s3a.connection.timeout as 5000, and the code
> under hadoop-tools/hadoop-aws defines it as 50000.
> We should update the former to 50s so it gets taken properly, as we're also
> noticing that 5s is often too low, especially in cases such as large DistCp
> operations (which fail with {{Read timed out}} errors from the S3 service).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
