> On Sept. 12, 2012, 11:37 p.m., Travis Crawford wrote: > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/mapreduce/FileRecordWriterContainer.java, > > line 203 > > <https://reviews.apache.org/r/7068/diff/1/?file=153057#file153057line203> > > > > It would be helpful for this comment to include a note that > > checkOutputSpecs is the call that creates the directory. Here we mention it > > could be created by another tasks, but I think we could clarify that the > > first task to reach this try block is what creates the directory. > > Arup Malakar wrote: > checkOutputSpecs doesn't create the directory, it just verifies that the > output directory doesn't already exist. It makes sense to do so in > FileOutputFormat. But the same check probably doesn't make sense in a > RecordWriter. Looking at the documentation I guess we could altogether remove > the call from FileRecordWriterContainer, let me know your opinion. > > http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/mapred/FileOutputFormat.java#FileOutputFormat.checkOutputSpecs%28org.apache.hadoop.fs.FileSystem%2Corg.apache.hadoop.mapred.JobConf%29
I believe you're right – this check doesn't make sense with dynamic partitioning. If you remove this check, the tests pass, and no wiser Hadoop Guru warns us otherwise, I agree we should remove the check. - Travis ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7068/#review11437 ----------------------------------------------------------- On Sept. 14, 2012, 5:40 p.m., Arup Malakar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7068/ > ----------------------------------------------------------- > > (Updated Sept. 14, 2012, 5:40 p.m.) > > > Review request for hcatalog and Travis Crawford. > > > Description > ------- > > 1. Added a test case to reproduce the bug > 2. Fixed the bug > 2. Apart from some cosmetic changes for readability in HCatMapReduceTest > Removed the abstract method initialize() in HCatMapReduceTest as the > same could be achieved by using the well understood method setUp() > > > This addresses bug HCATALOG-490. > https://issues.apache.org/jira/browse/HCATALOG-490 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/java/org/apache/hcatalog/mapreduce/FileRecordWriterContainer.java > 1384595 > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/test/org/apache/hcatalog/mapreduce/HCatBaseTest.java > 1384595 > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/test/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java > 1384595 > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/test/org/apache/hcatalog/mapreduce/TestHCatDynamicPartitioned.java > 1384595 > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/test/org/apache/hcatalog/mapreduce/TestHCatNonPartitioned.java > 1384595 > > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java > 1384595 > > Diff: https://reviews.apache.org/r/7068/diff/ > > > Testing > ------- > > test: > [mkdir] Created dir: > /Users/malakar/code/oss/hcatalog/trunk/build/test/logs > [mkdir] Created dir: > /Users/malakar/code/oss/hcatalog/trunk/build/test/hcat_junit_warehouse > [junit] WARNING: multiple versions of ant detected in path for junit > [junit] > jar:file:/Users/malakar/bin/apache-ant-1.8.4/lib/ant.jar!/org/apache/tools/ant/Project.class > [junit] and > jar:file:/Users/malakar/code/oss/hcatalog/trunk/build/ivy/lib/test/ant-ant-1.6.5.jar!/org/apache/tools/ant/Project.class > [junit] Running org.apache.hcatalog.cli.TestPermsGrp > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 14.854 sec > [junit] Running org.apache.hcatalog.cli.TestSemanticAnalysis > [junit] Tests run: 22, Failures: 0, Errors: 0, Time elapsed: 20.615 sec > [junit] Running org.apache.hcatalog.cli.TestUseDatabase > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 13.533 sec > [junit] Running org.apache.hcatalog.common.TestHCatUtil > [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.868 sec > [junit] Running org.apache.hcatalog.common.TestHiveClientCache > [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 38.042 sec > [junit] Shutting down hive metastore. > [junit] Running org.apache.hcatalog.data.TestDefaultHCatRecord > [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.21 sec > [junit] Running org.apache.hcatalog.data.TestHCatRecordSerDe > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.181 sec > [junit] Running org.apache.hcatalog.data.TestJsonSerDe > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.239 sec > [junit] Running org.apache.hcatalog.data.TestLazyHCatRecord > [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 0.177 sec > [junit] Running org.apache.hcatalog.data.TestReaderWriter > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.083 sec > [junit] Running org.apache.hcatalog.data.schema.TestHCatSchema > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.007 sec > [junit] Running org.apache.hcatalog.data.schema.TestHCatSchemaUtils > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.043 sec > [junit] Running org.apache.hcatalog.listener.TestMsgBusConnection > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 7.588 sec > [junit] Running org.apache.hcatalog.listener.TestNotificationListener > [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 8.045 sec > [junit] Test org.apache.hcatalog.listener.TestNotificationListener FAILED > [junit] Running org.apache.hcatalog.mapreduce.TestHCatDynamicPartitioned > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 122.809 sec > [junit] Running org.apache.hcatalog.mapreduce.TestHCatHiveCompatibility > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 32.425 sec > [junit] Running > org.apache.hcatalog.mapreduce.TestHCatHiveThriftCompatibility > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 11.662 sec > [junit] Running org.apache.hcatalog.mapreduce.TestHCatInputFormat > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 10.985 sec > [junit] Running org.apache.hcatalog.mapreduce.TestHCatMultiOutputFormat > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 28.85 sec > [junit] Shutting down hive metastore. > [junit] Running org.apache.hcatalog.mapreduce.TestHCatNonPartitioned > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 14.169 sec > [junit] Running org.apache.hcatalog.mapreduce.TestHCatOutputFormat > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.323 sec > [junit] Running org.apache.hcatalog.mapreduce.TestHCatPartitioned > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 92.827 sec > [junit] Running org.apache.hcatalog.mapreduce.TestMultiOutputFormat > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 67.213 sec > [junit] Running org.apache.hcatalog.mapreduce.TestPassProperties > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 8.611 sec > [junit] Running org.apache.hcatalog.mapreduce.TestSequenceFileReadWrite > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 53.583 sec > [junit] Running org.apache.hcatalog.rcfile.TestRCFileMapReduceInputFormat > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.216 sec > [junit] Running org.apache.hcatalog.security.TestHdfsAuthorizationProvider > [junit] Tests run: 34, Failures: 0, Errors: 0, Time elapsed: 23.476 sec > > > Thanks, > > Arup Malakar > >
