----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6219/ -----------------------------------------------------------
(Updated Aug. 1, 2012, 11:34 p.m.) Review request for hcatalog. Changes ------- -- Addressed rohini's comments except the one about move tasks. I would address that as a separate JIRA. Description ------- If an MR job using HCatOutputFormat fails, and FileOutputCommitterContainer::abortJob() is called, one would expect that partitions aren't created/registered with HCatalog. When using dynamic-partitions, one sees that this behaves correctly. But when static-partitions are used, partitions are created regardless of whether the Job succeeded or failed. (This manifested as a failure when the job is repeated. The retry-job fails to launch since the partitions already exist from the last failed run.) This is a result of bad code in FileOutputCommitter::cleanupJob(), which seems to do an unconditional partition-add. This can be fixed by adding a check for the output directory before adding partitions (in the !dynamicParititoning case), since the directory is removed in abortJob(). We'll have a patch for this shortly. As an aside, we ought to move the partition-creation into commitJob(), where it logically belongs. cleanupJob() is deprecated and common to both success and failure code paths. Diffs (updated) ----- /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java 1367361 /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java 1367361 /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InternalUtil.java 1367361 /branches/branch-0.4/src/java/org/apache/hcatalog/pig/HCatStorer.java 1367361 /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatOutputFormat.java 1367361 /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitionPublish.java PRE-CREATION /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestSequenceFileReadWrite.java 1367361 /branches/branch-0.4/src/test/org/apache/hcatalog/pig/TestHCatStorer.java 1367361 Diff: https://reviews.apache.org/r/6219/diff/ Testing ------- unit tests and e2e test pass. Thanks, Vandana Ayyalasomayajula
