----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7386/#review12102 -----------------------------------------------------------
/trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/7386/#comment25782> why make the state implicit as failed, why not pass the state instead? /trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/7386/#comment25787> add partitions should be done after the partitions have been moved to prevent metadata inconsistency. /trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/7386/#comment25791> This is no longer relevant for this patch? /trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/7386/#comment25789> Not sure why we need an internalAbortJob() can we move this code into abortJob()? /trunk/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java <https://reviews.apache.org/r/7386/#comment25792> There are a lot of format changes here and it looks like the only relevant change is the removal of a method. Can you update this file so only that change is made? - Francis Liu On Oct. 2, 2012, 1:09 a.m., Vandana Ayyalasomayajula wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7386/ > ----------------------------------------------------------- > > (Updated Oct. 2, 2012, 1:09 a.m.) > > > Review request for hcatalog and Francis Liu. > > > Description > ------- > > If an MR job using HCatOutputFormat fails, and > FileOutputCommitterContainer::abortJob() is called, one would expect that > partitions aren't created/registered with HCatalog. > When using dynamic-partitions, one sees that this behaves correctly. But when > static-partitions are used, partitions are created regardless of whether the > Job succeeded or failed. > (This manifested as a failure when the job is repeated. The retry-job fails > to launch since the partitions already exist from the last failed run.) > This is a result of bad code in FileOutputCommitter::cleanupJob(), which > seems to do an unconditional partition-add. This can be fixed by adding a > check for the output directory before adding partitions (in the > !dynamicParititoning case), since the directory is removed in abortJob(). > As an aside, we ought to move the partition-creation into commitJob(), where > it logically belongs. cleanupJob() is deprecated and common to both success > and failure code paths. > > > This addresses bug hcatalog-451. > https://issues.apache.org/jira/browse/hcatalog-451 > > > Diffs > ----- > > > /trunk/hcatalog-pig-adapter/src/test/java/org/apache/hcatalog/pig/TestHCatStorer.java > 1392658 > > /trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java > 1392658 > /trunk/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java 1392658 > /trunk/src/java/org/apache/hcatalog/mapreduce/InternalUtil.java 1392658 > /trunk/src/test/org/apache/hcatalog/mapreduce/TestHCatOutputFormat.java > 1392658 > /trunk/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitionPublish.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/7386/diff/ > > > Testing > ------- > > unit tests and e2e tests pass. > > > Thanks, > > Vandana Ayyalasomayajula > >
