[
https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225277#comment-16225277
]
Steve Loughran commented on HADOOP-14971:
-----------------------------------------
Had a conf call w/ Aaron on friday. He will have some fuller comments to make;
I've got some (from memory) notes.
* Proposed: restore different config options for s3a retry policy and aws SDK
policy, so that if s3 wraps AWS it doesn't double the retries. Issue: we aren't
100% sure when AWS sdk retries, though we know that it does for PUT of files,
and (in the past) does not for the final POST of an MPU. Action: restore the
options.
* Some discussion about how to react to NPEs &c in the AWS sdk. For now, do not
attempt to retry.
* What should we do about delete and idempotency? This is a long standing topic
of past HDFS discussion (HDFS-4872). Aaron: what did we conclude here?
* Sometimes DDB throws resource not found, esp. on test setup after a new table
is created. We way want to treat the DDB exception as retriable, even though S3
404 -> FNFE -> unrecoverable. (steve says: good idea, let's do it after the
merge)
* Security: we should document the security implications for the committers &
have it reviewed. Things to worry about: Denial of service,
destroying/tampering with data by altering .pendingset files; aborting pending
uploads. The Staging committers use the shared HDFS FS, so need to have
security policy there reviewed and documented. I'm reviewing how we use &
document this path so that it defaults to being something under the user's home
directory. That way: if HDFS is locked down, the path should be too
> Merge S3A committers into trunk
> -------------------------------
>
> Key: HADOOP-14971
> URL: https://issues.apache.org/jira/browse/HADOOP-14971
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>
> Merge the HADOOP-13786 committer into trunk. This branch is being set up as a
> github PR for review there & to keep it out the mailboxes of the watchers on
> the main JIRA
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
