I am using sqoop-1.4.4 because it integrated hcatalog into it
(https://issues.apache.org/jira/browse/SQOOP-931). Unfortunately, I cannot get
it to work because the HCatlog Incompatible issue. ( I have Hadoop
2.0.0-cdh4.3.0 installed)
First, I got
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was
expected
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:94)
It is expecting Hadoop 1.0.x but I have Hadoop 2.0.x. I took advice from Sqoop
developer Venkat Ranganathan and built Hcatalog with Hadoop 2.0 artifact and
replaced those jars under /usr/lib/hive/hcatalog/share/hcatalog and webhcat
with jars newly built.
The error went away. I created a transaction3 table using hcat first then ran
sqoop import --connect jdbc:mysql://<host>/<database> --username <user>
--password <password> --table transaction3 --where "date >= '2013-08-12'"
--split-by date --hcatalog-table transaction3
I got the following errors.
2013-08-18 12:08:23,087 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1376849374287_0002_m_000002_3: Error: java.lang.RuntimeException:
Failed to instantiate: org.apache.hcatalog.shims.HCatHadoopShims23
at
org.apache.hcatalog.shims.HCatHadoopShims$Instance.selectShim(HCatHadoopShims.java:67)
at
org.apache.hcatalog.shims.HCatHadoopShims$Instance.<clinit>(HCatHadoopShims.java:49)
at
org.apache.hadoop.mapred.HCatMapRedUtil.createTaskAttemptContext(HCatMapRedUtil.java:39)
at
org.apache.hadoop.mapred.HCatMapRedUtil.createTaskAttemptContext(HCatMapRedUtil.java:29)
:
Caused by: java.lang.ClassNotFoundException:
org.apache.hcatalog.shims.HCatHadoopShims23
My hcatalog-core-0.11.0.jar has HCatHadoopShims20S instead.
I read source code HCatHadoopShims. It have codes like the followings, Is any
way I can get this to work (override major version etc.)? Any help would be
appreciated? THANKS.
private static HCatHadoopShims selectShim() {
// piggyback on Hive's detection logic
String major = ShimLoader.getMajorVersion();
String shimFQN = "org.apache.hcatalog.shims.HCatHadoopShims20S";
if (major.startsWith("0.23")) {
shimFQN = "org.apache.hcatalog.shims.HCatHadoopShims23";
}
try {
Class<? extends HCatHadoopShims> clasz = Class.forName(shimFQN)
.asSubclass(HCatHadoopShims.class);
return clasz.newInstance();
} catch (Exception e) {
throw new RuntimeException("Failed to instantiate: " + shimFQN,
e);
}
}
