----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6219/#review9662 -----------------------------------------------------------
/branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20583> Its existing code. But we need to check if is there any reason why baseCommitter commitJob is called only for static partitions and not dynamic partitions. /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20587> This should go in the finally block of commitJob and abortJob. Can you also add a log message to Security.cancelToken saying cancelled delegation token. /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20586> Can we log the list of new columns being added to the table /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20585> Can we log the list of partitions that are going to be added /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20584> Shouldn't we move data before doing addPartitions similar to the if block? /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20580> Need to move token cancellation to a separate method and call it from finally block of both commitJob and abortJob. Can use the Security class's cancelToken method /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java <https://reviews.apache.org/r/6219/#comment20581> Just a little more formal message like "Job failed. So cleaning up temporary directory" + src /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitionPublish.java <https://reviews.apache.org/r/6219/#comment20578> Can you add one more assert to check if the temp directory is deleted by the abortJob /branches/branch-0.4/src/test/org/apache/hcatalog/pig/TestHCatStorer.java <https://reviews.apache.org/r/6219/#comment20579> One more assert to check if temp dir is deleted. - Rohini Palaniswamy On July 31, 2012, 1 a.m., Vandana Ayyalasomayajula wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6219/ > ----------------------------------------------------------- > > (Updated July 31, 2012, 1 a.m.) > > > Review request for hcatalog. > > > Description > ------- > > > If an MR job using HCatOutputFormat fails, and > FileOutputCommitterContainer::abortJob() is called, one would expect that > partitions aren't created/registered with HCatalog. > > When using dynamic-partitions, one sees that this behaves correctly. But when > static-partitions are used, partitions are created regardless of whether the > Job succeeded or failed. > (This manifested as a failure when the job is repeated. The retry-job fails > to launch since the partitions already exist from the last failed run.) > > This is a result of bad code in FileOutputCommitter::cleanupJob(), which > seems to do an unconditional partition-add. This can be fixed by adding a > check for the output directory before adding partitions (in the > !dynamicParititoning case), since the directory is removed in abortJob(). > > We'll have a patch for this shortly. As an aside, we ought to move the > partition-creation into commitJob(), where it logically belongs. cleanupJob() > is deprecated and common to both success and failure code paths. > > > Diffs > ----- > > > /branches/branch-0.4/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java > 1367361 > /branches/branch-0.4/src/java/org/apache/hcatalog/pig/HCatStorer.java > 1367361 > > /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatOutputFormat.java > 1367361 > > /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitionPublish.java > PRE-CREATION > > /branches/branch-0.4/src/test/org/apache/hcatalog/mapreduce/TestSequenceFileReadWrite.java > 1367361 > /branches/branch-0.4/src/test/org/apache/hcatalog/pig/TestHCatStorer.java > 1367361 > > Diff: https://reviews.apache.org/r/6219/diff/ > > > Testing > ------- > > unit tests and e2e test pass. > > > Thanks, > > Vandana Ayyalasomayajula > >
