VenkatSNarayanan commented on code in PR #339:
URL: https://github.com/apache/tez/pull/339#discussion_r1678189348
##########
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/MRJobConfig.java:
##########
@@ -131,6 +131,11 @@ public interface MRJobConfig {
public static final String CACHE_ARCHIVES_VISIBILITIES =
"mapreduce.job.cache.archives.visibilities";
+ /**
+ * Used by committers to set a job-wide UUID.
+ */
+ public static final String JOB_COMMITTER_UUID = "job.committer.uuid";
Review Comment:
@abstractdog I haven't publicly posted the Hadoop PR yet, but the change I
have is to check for this property around here:
https://github.com/apache/hadoop/blob/51cb858cc8c23d873d4adfc21de5f2c1c22d346f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitter.java#L1372
similar to how the Spark property is checked. I have tested these changes
together already alongside my Hive implementation.
@zhangbutao There are some corresponding changes to Hadoop and Hive that
also need to be merged which I have. Once all 3 PRs(Tez, Hadoop and Hive have
been merged), then the magic committer will be usable with Hive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
