-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6219/
-----------------------------------------------------------
(Updated Aug. 1, 2012, 11:35 p.m.)
Review request for hcatalog.
Description
-------
If an MR job using HCatOutputFormat fails, and
FileOutputCommitterContainer::abortJob() is called, one would expect that
partitions aren't created/registered with HCatalog.
When using dynamic-partitions, one sees that this behaves correctly. But when
static-partitions are used, partitions are created regardless of whether the
Job succeeded or failed.
(This manifested as a failure when the job is repeated. The retry-job fails to
launch since the partitions already exist from the last failed run.)
This is a result of bad code in FileOutputCommitter::cleanupJob(), which seems
to do an unconditional partition-add. This can be fixed by adding a check for
the output directory before adding partitions (in the !dynamicParititoning
case), since the directory is removed in abortJob().
We'll have a patch for this shortly. As an aside, we ought to move the
partition-creation into commitJob(), where it logically belongs. cleanupJob()
is deprecated and common to both success and failure code paths.
This addresses bug hcatalog-451.
https://issues.apache.org/jira/browse/hcatalog-451
Diffs
-----
/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java
1367361
/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java
1367361
/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/InternalUtil.java
1367361
/branches/branch-0.4/src/java/org/apache/hcatalog/pig/HCatStorer.java 1367361
/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatOutputFormat.java
1367361
/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitionPublish.java
PRE-CREATION
/branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestSequenceFileReadWrite.java
1367361
/branches/branch-0.4/src/test/org/apache/hcatalog/pig/TestHCatStorer.java
1367361
Diff: https://reviews.apache.org/r/6219/diff/
Testing
-------
unit tests and e2e test pass.
Thanks,
Vandana Ayyalasomayajula
