[
https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442215#comment-13442215
]
Dmitriy V. Ryaboy commented on PIG-2885:
----------------------------------------
Great, I think we are getting close.
I still don't like manually adding hbase's dependencies. Pretty sure we can add
configurations to our ivy and make a similar distinction to HBase's, and depend
on hbase compile->runtime in test, but on compile->master for the normal build.
Also, we should update all the docs that mention compatibility and mention
somewhere that one needs to set HBASE_CLASSPATH.
I noticed that hadoop version bump. Is that required for hbase, or just
something you took care of while you were in there? I'm not sure what we want
to do about minor hadoop version compatibility, but just thought I'd bring that
up...
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by
> bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop
> 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml -
> high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer
> 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
> "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
> Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that
> TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
