This is a "me too" post: we just ran into an identical problem setting
up a new cluster using CDH3b2. This is all rather mystifying, because
all the correct libraries are there; in fact the hive command line
looks something like
/usr/java/jdk1.6.0_12/bin/java -Xmx256m -server
-Dhadoop.log.dir=/usr/lib/hadoop-0.20/logs
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Dhadoop.policy.file=hadoop-policy.xml -classpath
/usr/lib/hadoop-0.20/conf:/usr/java/jdk1.6.0_12/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2+320.jar..................
org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-cli-0.5.0.jar
org.apache.hadoop.hive.cli.CliDriver
(note the presence of the hadoop-core.jar which contains the class in question)
If anyone has pointers on troubleshooting this, I'd appreciate it!
--Leo
On Tue, Sep 14, 2010 at 1:56 PM, Tianqiang Li <peter...@gmail.com> wrote:
> Some more context: I run Hive on a client machine which is NOT one of the
> hadoop cluster nodes, I suppose Hive can run well and submit job from a
> client machine, so I didn't change hadoop-env.sh on cluster nodes. On this
> client machine, hadoop java jobs and pig have been successfully submitted to
> cluster and processed.
>
> Regards,
> Peter Li
>
> On Tue, Sep 14, 2010 at 1:42 PM, Tianqiang Li <peter...@gmail.com> wrote:
>>
>> Hi, hive-users,
>> I am a new Hive users, install hive recently, when I type any
>> query-related command in Hive cli, it throws the exception, but create table
>> are ok:
>>
>> $ hive
>> Hive history file=/tmp/pli/hive_job_log_pli_201009141519_1503313446.txt
>> hive> create table test5(a int);
>> OK
>> Time taken: 2.551 seconds
>> hive> show tables;
>> OK
>> Failed with exception java.io.IOException:java.io.IOException: Cannot
>> create an instance of InputFormat class
>> org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
>> Time taken: 0.09 seconds
>> hive>
>>
>> I searched on internet, and there are some info about hive's
>> HADOOP_CLASSPATH getting overwritten by some other tools in hadoop-env.sh, I
>> tried to use appending( export
>> HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:other_paths ), didn't fix the problem,
>> then I tried to copy the jar files of hadoop libraries (
>> hadoop-0.20.1+169.68-core.jar ) which include
>> org.apache.hadoop.mapred.TextInputFormat, to $HIVE_HOME/lib/, still the
>> problem remain the same, It looks like Hive doesn't know where hadoop's
>> libraries are, and failed to new a TextInputFormat object, are there someone
>> met this before? Any hints on how to work this around are welcome. Thanks.
>>
>> Here are some more context, I use hadoop 0.20.1+169.68 from cloudera CDH2,
>> and Hive 0.4/0.5 from CDH2/3, (both version for hive have the same issues),
>> here is the trace stack in log file.
>> ------
>> 2010-09-14 15:19:58,826 ERROR DataNucleus.Plugin
>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>> "org.eclipse.core.resources" but it cannot be resolved.
>> 2010-09-14 15:19:58,826 ERROR DataNucleus.Plugin
>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>> "org.eclipse.core.resources" but it cannot be resolved.
>> 2010-09-14 15:19:58,828 ERROR DataNucleus.Plugin
>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>> "org.eclipse.core.runtime" but it cannot be resolved.
>> 2010-09-14 15:19:58,828 ERROR DataNucleus.Plugin
>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>> "org.eclipse.core.runtime" but it cannot be resolved.
>> 2010-09-14 15:19:58,829 ERROR DataNucleus.Plugin
>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>> "org.eclipse.text" but it cannot be resolved.
>> 2010-09-14 15:19:58,829 ERROR DataNucleus.Plugin
>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
>> "org.eclipse.text" but it cannot be resolved.
>> 2010-09-14 15:20:04,444 ERROR CliDriver
>> (SessionState.java:printError(279)) - Failed with exception
>> java.io.IOException:java.io.IOException: Cannot create an instance of
>> InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in
>> mapredWork!
>> java.io.IOException: java.io.IOException: Cannot create an instance of
>> InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in
>> mapredWork!
>> at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:273)
>> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
>> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:440)
>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:131)
>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.io.IOException: Cannot create an instance of InputFormat
>> class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!
>> at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:113)
>> at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:214)
>> at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:249)
>>
>> ----
>> My hadoop-env.sh :
>> export
>> HADOOP_CLASSPATH_OTHER=/usr/lib/hadoop-0.20/hadoop-0.20.1+169.68-core.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/*:/usr/lib/hadoop-0.20/*
>> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}$:{HADOOP_CLASSPATH_OTHER}
>>
>>
>>
>>
>> Regards,
>> Peter Li
>
>
