[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442194#comment-13442194 ]
Cheolsoo Park commented on PIG-2885: ------------------------------------ Hi Dmitriy, they're defined in hbase's pom, but they are not pulled down by pig's ivy because the dependency is defined as follows: {code} <dependency org="org.apache.hbase" name="hbase" rev="${hbase.version}" conf="compile->master"> {code} As can be seen, the "compile" configuration depends on "master" not "runtime", so runtime dependencies such as protobuf and high-scale-lib are not pulled down at compile time. I could change "master" to "runtime" instead of adding separate dependency tags, but then we would pull down unnecessary dependencies at compile time even though they're only needed at runtime. Thanks! > TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3 > ----------------------------------------------------------------------------- > > Key: PIG-2885 > URL: https://issues.apache.org/jira/browse/PIG-2885 > Project: Pig > Issue Type: Bug > Environment: Hadoop 1.0.3, CentOS 6.3 64 bit > Reporter: Cheolsoo Park > Assignee: Cheolsoo Park > Priority: Minor > Attachments: PIG-2885.patch > > > I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by > bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop > 1.0.3. I am opening a jira to capture what I found for future reference. > - Two dependency libraries of HBase 0.94 are missing in ivy.xml - > high-scale-lib and protobuf-java. > - The HTable constructor in HBase 0.94 changed: > {code} > - HTable table = new HTable(TESTTABLE_2); > + HTable table = new HTable(conf, TESTTABLE_2); > {code} > - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer > 21818. Since it is chosen randomly at runtime, it has to be set in PigContext. > {code} > @@ -541,7 +543,7 @@ public class TestJobSubmission { > // use the estimation > Configuration conf = cluster.getConfiguration(); > HBaseTestingUtility util = new HBaseTestingUtility(conf); > - util.startMiniZKCluster(); > + int clientPort = util.startMiniZKCluster().getClientPort(); > util.startMiniHBaseCluster(1, 1); > > String query = "a = load '/passwd';" + > @@ -553,6 +555,7 @@ public class TestJobSubmission { > > pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", > "100"); > pc.getConf().setProperty("pig.exec.reducers.max", "10"); > + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, > Integer.toString(clientPort)); > ConfigurationValidator.validatePigProperties(pc.getProperties()); > conf = ConfigurationUtil.toConfiguration(pc.getProperties()); > JobControlCompiler jcc = new JobControlCompiler(pc, conf); > {code} > With the attached patch, both tests pass with hadoop 1.0.3. Please note that > TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira