[
https://issues.apache.org/jira/browse/HADOOP-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469191
]
Tom White commented on HADOOP-882:
----------------------------------
The best way to check the new jets3t library is to run the
Jets3tS3FileSystemTest unit test. You will need to set your S3 credentials in
the hadoop-site.xml file in the test directory. If this passes you can be
confident that the upgrade has worked.
Changing the buffer size to be as big as the block size sounds good (however, I
worry a little whether there could be a memory issue if jets3t buffers in
memory, as seems likely).
The 1MB block size was a fairly arbitrary value I selected during testing. I
agree that 64MB would be better. The property fs.s3.block.size property needs
adding to hadoop-default.xml, and the DEFAULT_BLOCK_SIZE constant in
S3FileSystem needs changing too.
A patch for all this stuff would be very welcome!
As for whether completing all these items would mean the issue is closed I'm
not sure. The jets3t retry mechanism is for S3-level exceptions, if there is a
traffic-level communication problem then there is nothing to handle it. For
this, we could use the more general Hadoop-level mechanism, described above (or
possibly use the retry-mechanism in HttpClient, if that's sufficient). I think
this work would belong in another Jira issue. Thoughts?
> S3FileSystem should retry if there is a communication problem with S3
> ---------------------------------------------------------------------
>
> Key: HADOOP-882
> URL: https://issues.apache.org/jira/browse/HADOOP-882
> Project: Hadoop
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.10.1
> Reporter: Tom White
> Assigned To: Tom White
>
> File system operations currently fail if there is a communication problem
> (IOException) with S3. All operations that communicate with S3 should retry a
> fixed number of times before failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
