[
https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-2885:
-------------------------------
Release Note: This makes hbase.jar and zookeeper.jar no longer be embedded
in pig.jar, and thus, to use HBaseStorage with pig.jar, hbase.jar and
zookeeper.jar must be present in classpath. To add them to classpath, the user
can set either PIG_CLASSPATH=<path_to_hbase.jar:path_to_zookeeper.jar>, or
HBASE_HOME=<hbase_home_dir> and ZOOKEEPER_HOME=<zookeeper_home_dir>.
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885-3.patch, PIG-2885-4.patch,
> PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by
> bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop
> 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml -
> high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer
> 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
> "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
> Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that
> TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
