[
https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-2885:
-------------------------------
Attachment: PIG-2885-2.patch
Updating the patch:
1. Stop bundling hbase.jar and zookeoper.jar with pig.jar. So there should be
no longer incompatibility issues when using pig.jar with different versions of
hbase.jar. But to use HBaseStorage, HBASE_HOME and ZOOKEEPER_HOME must be set
by the user.
Note that I am adding protobuf-java.jar to pig.jar because otherwise it has to
be explicitly added to PIG_CLASSPATH to use HBaseStorage, which is not very
intuitive.
2. Bump hbase and zk to 0.94.1 and 3.4.3 respectively. Since we no longer
bundle them in pig.jar, which versions we use doesn't matter. These jar files
will be used for unit test only.
3. Make the unit test cases work with newer versions of hbase and zk.
4. Add hbase runtime dependencies to ivy.xml. I tried to find an alternative
way of specifying distinct dependencies in ivy.xml without adding them
explicitly but no luck.
Basically, what I want to do is as follows:
- pulls down hbase.jar only at compile time.
- pulls down runtime dependencies as well at test time.
What I tried was "conf=compile->master;test->default", but apparently, it
doesn't work. Instead, runtime depedencies are always pulled down. Please let
me know if anyone has a better idea about how to achieve this.
Thanks!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by
> bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop
> 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml -
> high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer
> 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
> "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
> Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that
> TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
