Mithun Radhakrishnan created HCATALOG-451:
---------------------------------------------
Summary: Partitions are created even when Jobs are aborted
Key: HCATALOG-451
URL: https://issues.apache.org/jira/browse/HCATALOG-451
Project: HCatalog
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.4
Environment: Hadoop 1.0.2, non-dynamic partitions.
Reporter: Mithun Radhakrishnan
Fix For: 0.4.1
If an MR job using HCatOutputFormat fails, and
FileOutputCommitterContainer::abortJob() is called, one would expect that
partitions aren't created/registered with HCatalog.
When using dynamic-partitions, one sees that this behaves correctly. But when
static-partitions are used, partitions are created regardless of whether the
Job succeeded or failed.
(This manifested as a failure when the job is repeated. The retry-job fails to
launch since the partitions already exist from the last failed run.)
This is a result of bad code in FileOutputCommitter::cleanupJob(), which seems
to do an unconditional partition-add. This can be fixed by adding a check for
the output directory before adding partitions (in the !dynamicParititoning
case), since the directory is removed in abortJob().
We'll have a patch for this shortly. As an aside, we ought to move the
partition-creation into commitJob(), where it logically belongs. cleanupJob()
is deprecated and common to both success and failure code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
