Hi Prasad,
I'm running java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Server VM (build 11.0-b16, mixed mode)
I don't have a $HADOOP_CLASSPATH environment variable nor is hadoop in
my path. The only hadoop environment variable that I have set is
$HADOOP_HOME, which seems to be required by the hive script.
The output that I get from running the hive script is:
-bash-3.2$ ./hive
/users/scohen/hive/build/dist/bin/../conf:/users/scohen/hive/build/dist/bin/../lib/TestSerDe.jar:/users/scohen/hive/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/users/scohen/hive/build/dist/bin/../lib/asm-3.1.jar:/users/scohen/hive/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/users/scohen/hive/build/dist/bin/../lib/commons-collections-3.2.1.jar:/users/scohen/hive/build/dist/bin/../lib/commons-jexl-1.1.jar:/users/scohen/hive/build/dist/bin/../lib/commons-lang-2.4.jar:/users/scohen/hive/build/dist/bin/../lib/commons-logging-1.0.4.jar:/users/scohen/hive/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/users/scohen/hive/build/dist/bin/../lib/derby.jar:/users/scohen/hive/build/dist/bin/../lib/hive_anttasks.jar:/users/scohen/hive/build/dist/bin/../lib/hive_cli.jar:/users/scohen/hive/build/dist/bin/../lib/hive_common.jar:/users/scohen/hive/build/dist/bin/../lib/hive_exec.jar:/users/scohen/hive/build/dist/bin/../lib/hive_jdbc.jar:/users/scohen/hive/build/dist/bin/../lib/hive_metastore.jar:/users/scohen/hive/build/dist/bin/../lib/hive_serde.jar:/users/scohen/hive/build/dist/bin/../lib/hive_service.jar:/users/scohen/hive/build/dist/bin/../lib/jdo2-api-2.1.jar:/users/scohen/hive/build/dist/bin/../lib/jline-0.9.94.jar:/users/scohen/hive/build/dist/bin/../lib/jpox-core-1.2.2.jar:/users/scohen/hive/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/users/scohen/hive/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/users/scohen/hive/build/dist/bin/../lib/json.jar:/users/scohen/hive/build/dist/bin/../lib/junit-3.8.1.jar:/users/scohen/hive/build/dist/bin/../lib/libfb303.jar:/users/scohen/hive/build/dist/bin/../lib/libthrift.jar:/users/scohen/hive/build/dist/bin/../lib/log4j-1.2.15.jar:/users/scohen/hive/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/users/scohen/hive/build/dist/bin/../lib/velocity-1.5.jar:
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 7 more
I have echoed the last line, but the script errors out before reaching
the last line, so nothing gets printed out except for what I have copied
above.
Attached, is the hive script that I'm using (just in case that helps).
Please let me know if you have any idea what's going on. I agree that
this is weird!
Thanks,
Shirley
Prasad Chakka wrote:
Hi Shirley,
I am not sure what is happening here. Could you give us the following?
Java version (it should have java6)
HADOOP_CLASSPATH env value (should be nothing)
`which hadoop` (remove hadoop from PATH if it is there)
Echo the last line of bin/hive
------------------------------------------------------------------------
*From: *Shirley Cohen <sco...@cs.utexas.edu>
*Reply-To: *<hive-user@hadoop.apache.org>
*Date: *Thu, 29 Jan 2009 18:29:27 -0800
*To: *<hive-user@hadoop.apache.org>
*Subject: *Re: classpath issue
Yes, it's there. Here is the output from the jar command:
-bash-3.2$ ./jar tvf
/users/scohen/hive/build/dist/bin/../lib/hive_common.jar
0 Thu Jan 29 09:55:10 MST 2009 META-INF/
102 Thu Jan 29 09:55:08 MST 2009 META-INF/MANIFEST.MF
0 Thu Jan 29 09:55:10 MST 2009 org/
0 Thu Jan 29 09:55:10 MST 2009 org/apache/
0 Thu Jan 29 09:55:10 MST 2009 org/apache/hadoop/
0 Thu Jan 29 09:55:10 MST 2009 org/apache/hadoop/hive/
0 Thu Jan 29 09:55:10 MST 2009 org/apache/hadoop/hive/conf/
6447 Thu Jan 29 09:55:10 MST 2009
org/apache/hadoop/hive/conf/HiveConf$ConfVars.class
9123 Thu Jan 29 09:55:10 MST 2009
org/apache/hadoop/hive/conf/HiveConf.class
Thanks,
Shirley
Ashish Thusoo wrote:
> hive_common.jar is there on the classpath. Can you do the following
>
> jar tvf /users/scohen/hive/build/dist/bin/../lib/hive_common.jar
>
> and see if HiveConf shows up in that...
>
> Ashish
>
> -----Original Message-----
> From: Shirley Cohen [mailto:sco...@cs.utexas.edu]
> Sent: Thursday, January 29, 2009 2:26 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: classpath issue
>
> Great! Here is the print out of my classpath:
>
>
/users/scohen/hive/build/dist/bin/../conf:/users/scohen/hive/build/dist/bin/../lib/TestSerDe.jar:/users/scohen/hive/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/users/scohen/hive/build/dist/bin/../lib/asm-3.1.jar:/users/scohen/hive/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/users/scohen/hive/build/dist/bin/../lib/commons-collections-3.2.1.jar:/users/scohen/hive/build/dist/bin/../lib/commons-jexl-1.1.jar:/users/scohen/hive/build/dist/bin/../lib/commons-lang-2.4.jar:/users/scohen/hive/build/dist/bin/../lib/commons-logging-1.0.4.jar:/users/scohen/hive/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/users/scohen/hive/build/dist/bin/../lib/derby.jar:/users/scohen/hive/build/dist/bin/../lib/hive_anttasks.jar:/users/scohen/hive/build/dist/bin/../lib/hive_cli.jar:/users/scohen/hive/build/dist/bin/../lib/hive_common.jar:/users/scohen/hive/build/dist/bin/../lib/hive_exec.jar:/users/scohen/hive/build/dist/bin/../lib/hive_jdbc.jar:/users/scohen/hive/build/dist/bin/../lib/hive_metastore.jar:/users/scohen/hive/build/dist/bin/../lib/hive_serde.jar:/users/scohen/hive/build/dist/bin/../lib/hive_service.jar:/users/scohen/hive/build/dist/bin/../lib/jdo2-api-2.1.jar:/users/scohen/hive/build/dist/bin/../lib/jline-0.9.94.jar:/users/scohen/hive/build/dist/bin/../lib/jpox-core-1.2.2.jar:/users/scohen/hive/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/users/scohen/hive/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/users/scohen/hive/build/dist/bin/../lib/json.jar:/users/scohen/hive/build/dist/bin/../lib/junit-3.8.1.jar:/users/scohen/hive/build/dist/bin/../lib/libfb303.jar:/users/scohen/hive/build/dist/bin/../lib/libthrift.jar:/users/scohen/hive/build/dist/bin/../lib/log4j-1.2.15.jar:/users/scohen/hive/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/users/scohen/hive/build/dist/bin/../lib/velocity-1.5.jar:
>
> Please let me know your thoughts.
>
> Thanks,
>
> Shirley
>
> Prasad Chakka wrote:
>
>> You can change the last line bin/hive script to print the classpath,
>> may we can figure out from that.
>>
>>
>> ----------------------------------------------------------------------
>> --
>> *From: *Shirley Cohen <sco...@cs.utexas.edu>
>> *Reply-To: *<hive-user@hadoop.apache.org>
>> *Date: *Thu, 29 Jan 2009 11:04:23 -0800
>> *To: *<hive-user@hadoop.apache.org>
>> *Subject: *classpath issue
>>
>> Hi,
>>
>> I'm finally getting around to installing hive. I was able to build the
>> latest version from the repository without any problems. However, when
>> I try to run cli from from build/dist/bin, I get the following error:
>>
>> ./bin/hive -hiveconf hive.root.logger=INFO,console
>> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Unknown Source)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hive.conf.HiveConf
>> at java.net.URLClassLoader$1.run(Unknown Source)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(Unknown Source)
>> at java.lang.ClassLoader.loadClass(Unknown Source)
>> at java.lang.ClassLoader.loadClass(Unknown Source)
>> at java.lang.ClassLoader.loadClassInternal(Unknown Source)
>> ... 7 more
>>
>> From the error, it looks as though my classpath isn't getting set
>> correctly. Any ideas on why that might be?
>>
>> I'm running on FedoraCore 8 and hadoop 19. I have set the $HADOOP_HOME
>> variable to the root of my hadoop directory.
>>
>> Thanks,
>>
>> Shirley
>>
>>
>>
>
>
>
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
bin=`dirname "$0"`
bin=`cd "$bin"; pwd`
. "$bin"/hive-config.sh
CLASSPATH="${HIVE_CONF_DIR}"
HIVE_LIB=${HIVE_HOME}/lib
# needed for execution
if [ ! -f "${HIVE_LIB}/hive_exec.jar" ]; then
echo "Missing Hive Execution Jar: ${HIVE_LIB}/hive_exec.jar"
exit 1;
fi
if [ ! -f "${HIVE_LIB}/hive_metastore.jar" ]; then
echo "Missing Hive MetaStore Jar"
exit 2;
fi
# cli specific code
if [ ! -f "${HIVE_LIB}/hive_cli.jar" ]; then
echo "Missing Hive CLI Jar"
exit 3;
fi
for f in ${HIVE_LIB}/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
# add the auxillary jars such as serdes
if [ -d "${HIVE_AUX_JARS_PATH}" ]; then
for f in ${HIVE_AUX_JARS_PATH}/*.jar; do
if [[ ! -f $f ]]; then
continue;
fi
AUX_CLASSPATH=${AUX_CLASSPATH}:$f
if [ "${AUX_PARAM}" == "" ]; then
AUX_PARAM=file://$f
else
AUX_PARAM=${AUX_PARAM},file://$f;
fi
done
elif [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
AUX_CLASSPATH=${HIVE_AUX_JARS_PATH}
AUX_PARAM=file://${HIVE_AUX_JARS_PATH}
AUX_PARAM=`echo $AUX_PARAM | sed 's/,/,file:\/\//g'`
fi
# adding jars from auxlib directory
for f in ${HIVE_HOME}/auxlib/*.jar; do
if [[ ! -f $f ]]; then
continue;
fi
AUX_CLASSPATH=${AUX_CLASSPATH}:$f
if [ "${AUX_PARAM}" == "" ]; then
AUX_PARAM=file://$f
else
AUX_PARAM=${AUX_PARAM},file://$f;
fi
done
CLASSPATH=${CLASSPATH}:${AUX_CLASSPATH}
echo $CLASSPATH
# pass classpath to hadoop
export HADOOP_CLASSPATH=${CLASSPATH}
# check for hadoop in the path
HADOOP_IN_PATH=`which hadoop 2>/dev/null`
if [ -f ${HADOOP_IN_PATH} ]; then
HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
fi
# HADOOP_HOME env variable overrides hadoop in the path
HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
if [ "$HADOOP_HOME" == "" ]; then
echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or hadoop
must be in the path";
exit 4;
fi
HADOOP=$HADOOP_HOME/bin/hadoop
if [ ! -f ${HADOOP} ]; then
echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or hadoop
must be in the path";
exit 4;
fi
if [ "${AUX_PARAM}" != "" ]; then
HIVE_OPTS="$HIVE_OPTS -hiveconf hive.aux.jars.path=${AUX_PARAM}"
AUX_JARS_CMD_LINE="-libjars ${AUX_PARAM}"
fi
SERVICE_LIST=""
for i in "$bin"/ext/*.sh ; do
. $i
done
SERVICE=""
HELP=""
while [ $# -gt 0 ]; do
case "$1" in
--service)
shift
SERVICE=$1
shift
;;
--help)
HELP=_help
shift
;;
*)
break
;;
esac
done
if [ "$SERVICE" = "" ] ; then
if [ "$HELP" = "_help" ] ; then
SERVICE="help"
else
SERVICE="cli"
fi
fi
TORUN=""
for j in $SERVICE_LIST ; do
if [ "$j" = "$SERVICE" ] ; then
TORUN=${j}$HELP
fi
done
if [ "$TORUN" = "" ] ; then
echo "Service $SERVICE not found"
echo "Available Services: $SERVICE_LIST"
exit 5
else
$TORUN "$@"
fi
echo done
