[ https://issues.apache.org/jira/browse/HADOOP-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146008#comment-15146008 ]
Thomas Demoor commented on HADOOP-12353: ---------------------------------------- [~ste...@apache.org]: s3a does retry as the AWS SDK has a default exponential backoff retry policy that retries IO, HTTP 5xx, ..(http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#DEFAULT_RETRY_POLICY). We could make this configurable. [~raviprak]: You make a valid point but s3n is unmaintained code that's planned to be deprecated (people are focusing on s3a). A previous harmless-looking s3n patch broke things in an unexpected way hence I understand [~ste...@apache.org]'s point as well. He didn't -1 the patch, he simply doesn't want to be held responsible if it breaks things, but someone else might. > S3 Native filesystem does not retry all connection failures > ----------------------------------------------------------- > > Key: HADOOP-12353 > URL: https://issues.apache.org/jira/browse/HADOOP-12353 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 2.7.1 > Reporter: Mariusz Strzelecki > Assignee: Mariusz Strzelecki > Priority: Minor > Attachments: HADOOP-12353.001.patch, HADOOP-12353.002.patch > > > Current implementation of NativeS3FileSystem.java uses RetryProxy that > retries exceptions that may occur on network communication with S3 API, but > these exceptions must be exact instances of IOException: > https://github.com/apache/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java#L349 > Our tests show that HttpClient throws IOException subclasses which are not > handled by Proxy. > Additionally, not all methods that call S3 API are listed to be handled, i.e. > storeEmptyFile and retrieveMetadata are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)