Cheolsoo Park created PIG-2885:
----------------------------------
Summary: TestJobSumission and TestHBaseStorage don't work with
HBase 0.94 and ZK 3.4.3
Key: PIG-2885
URL: https://issues.apache.org/jira/browse/PIG-2885
Project: Pig
Issue Type: Bug
Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
Reporter: Cheolsoo Park
Priority: Minor
Attachments: PIG-2885.patch
I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by
bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop
1.0.3. I am opening a jira to capture what I found for future reference.
- Two dependency libraries of HBase 0.94 are missing in ivy.xml -
high-scale-lib and protobuf-java.
- The HTable constructor in HBase 0.94 changed:
{code}
- HTable table = new HTable(TESTTABLE_2);
+ HTable table = new HTable(conf, TESTTABLE_2);
{code}
- The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer
21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
{code}
@@ -541,7 +543,7 @@ public class TestJobSubmission {
// use the estimation
Configuration conf = cluster.getConfiguration();
HBaseTestingUtility util = new HBaseTestingUtility(conf);
- util.startMiniZKCluster();
+ int clientPort = util.startMiniZKCluster().getClientPort();
util.startMiniHBaseCluster(1, 1);
String query = "a = load '/passwd';" +
@@ -553,6 +555,7 @@ public class TestJobSubmission {
pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
pc.getConf().setProperty("pig.exec.reducers.max", "10");
+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
Integer.toString(clientPort));
ConfigurationValidator.validatePigProperties(pc.getProperties());
conf = ConfigurationUtil.toConfiguration(pc.getProperties());
JobControlCompiler jcc = new JobControlCompiler(pc, conf);
{code}
With the attached patch, both tests pass with hadoop 1.0.3. Please note that
TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
