[
https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442691#comment-13442691
]
Cheolsoo Park commented on PIG-2885:
------------------------------------
Actually, we have a bigger problem. The hbase jar in the maven repository is
not binary-compatible with hadoop 0.23 (HBASE-5680). To get hbase working with
hadoop 0.23, we have to recompile the source code against hadoop 0.23.
Using the hbase jar from the maven repository makes "TestHBaseStorage
-Dhadoopversion=23" fail with the following error:
{code}
2012-08-25 12:55:47,100 FATAL
[Master:0;localhost.localdomain,49603,1345924546912] master.HMaster
(HMaster.java:abort(1388)) - HBase is having a problem with its Hadoop jars.
You may need to recompile HBase against Hadoop version 0.23.1 or change your
hadoop jars to start properly
java.lang.NoClassDefFoundError:
org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:112)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:480)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:343)
at java.lang.Thread.run(Thread.java:662)
{code}
I am going to table this for now.
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by
> bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop
> 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml -
> high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer
> 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
> "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
> Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that
> TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
