Hi,
The documentation for the S3A committer [1] states that the Staging
Committer sets the `mapreduce.fileoutputcommitter.algorithm.version`
to "2" for that part of its work.
In `org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter.
initFileOutputCommitterOptions` [2] however we see
context.getConfiguration()
.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
1);
and the comment reads "In particular, this code currently only works
with commit algorithm 1.". The logs also suggest for us that it is set
to "1" by the log output
"FileOutputCommitter: File Output Committer Algorithm version is 1"
which seems to also be the case in test output in the corresponding
jira ticket [3] by Steve Laughran. Now my question is whether the
documentation is wrong, or otherwise what the docs are referring to?
Thanks,
Fabian
[1]
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/committers.html#FileOutputCommitter_appears_to_be_still_used_.28from_logs_or_delays_in_commits.29
[2]
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java
[3]
https://issues.apache.org/jira/browse/HADOOP-13786?focusedCommentId=15939015&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15939015
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
