[ https://issues.apache.org/jira/browse/HADOOP-19091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821619#comment-17821619 ]
Steve Loughran commented on HADOOP-19091: ----------------------------------------- [~vnarayanan7] * everything has to be done via github pRs right now; ;look at the hadoop-aws testing doc to see policies there. * all work to be on hadoop trunk with backports to 3.4.x: don't expect anything to go any earlier. * anything going near committers has to show why it is *correct* rather than just passing some tests. Correct ::= atomic and recoverable task commit, support for speculative tasks. job commit does not need atomicity though it's always nice. * and changes had better not break spark. I have put a lot of effort into not knowing anything about hive; I hope I don't have to change that policy. Having had a quick look at the patch: it would be better if the hive design used our committer factory design as spark does. That allows for different committees for different schemas, e.g. the manifest committer for abfs/gcs. it should also not contained hard coded checks for class types, with PathOutputCommitter the committer API to get working directories. > Add support for Tez to MagicS3GuardCommitter > -------------------------------------------- > > Key: HADOOP-19091 > URL: https://issues.apache.org/jira/browse/HADOOP-19091 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 > Affects Versions: 3.4.0, 3.3.6 > Environment: Pig 17/Hive 3.1.3 with Hadoop 3.3.3 on AWS EMR 6-12.0 > Reporter: Venkatasubrahmanian Narayanan > Priority: Major > Attachments: 0001-AWS-Hive-Changes.patch, > 0002-HIVE-27698-Backport-of-HIVE-22398-Remove-legacy-code.patch, > HADOOP-19091-HIVE-WIP.patch > > > The MagicS3GuardCommitter assumes that the JobID of the task is the same as > that of the job's application master when writing/reading the .pendingset > file. This assumption is not valid when running with Tez, which creates > slightly different JobIDs for tasks and the application master. > > While the MagicS3GuardCommitter is intended only for MRv2, it mostly works > fine with an MRv1 wrapper with Hive/Pig (with some minor changes to Hive) run > in MR mode. This issue only crops up when running queries with the Tez > execution engine. I can upload a patch to Hive 3.1 to reproduce this error on > EMR if needed. > > Fixing this will probably require work from both Tez and Hadoop, wanted to > start a discussion here so we can figure out how exactly we go about this. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org