[ https://issues.apache.org/jira/browse/HADOOP-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884156#comment-16884156 ]
Steve Loughran commented on HADOOP-16428: ----------------------------------------- I'm saying: its hard to show any benefits over the -direct option. It's what I do. Put differently: what problem to you have which the -direct option doesn't address? > Distcp don't make use of S3a Committers, be it magic or staging > --------------------------------------------------------------- > > Key: HADOOP-16428 > URL: https://issues.apache.org/jira/browse/HADOOP-16428 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, tools/distcp > Affects Versions: 3.1.1 > Reporter: Sahil Kaw > Priority: Minor > > Currently, I don't see Distcp make use of S3a Committers, be it Magic or > Staging and I have noticed most of the jobs which use MapReduce frameworks > use S3 committers except distcp. Distcp makes use of the FileOutputCommitter > even if S3a committer parameters are specified in the core-site.xml. Is this > by design? If yes, can someone please explain the reason for that. Are there > any limitations or potential risks of using S3a committers with Distcp? > I know there is a "-direct" option that can be used with the > FileOutputCommitter in order to avoid renaming while committing fr object > Stores. But if anyone can put some light on the current limitation of S3a > committers with distcp and reason for choosing FileOutputCommitters for > Distcp over S3a committers, it would be helpful. Thanks -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org