I ended up patching the HiveConf.java . If hive-site.xml were not found on the classpath then:
- an o.a.h.fs.Path object is created from System.getenv("HIVE_CONF_DIR") + File.seperator + "hive-site.xml" - the Path is sent to the base class Configuration.addResource - whiich btw accepts either resourceURL's or Path objects the problem was resolved with that change to HiveConf.java: the hive-site.xml is loaded properly via the Path (instead of URL) and as a consequence the jdoConnectionURL and related properties for the mysql metastore are loaded allowing the metadata to be stored in mysql 2012/12/9 Stephen Boesch <java...@gmail.com> > The first element of the classpath is the right one already.. but I STILL > get the hive-site.xml is not found in classpath. Only hive gives me > issues. hdfs, mapred, hbase are all running fine. > > HADOOP_CLASSPATH=:*/shared/hive/conf:* > /shared/hive/lib/antlr-runtime-3.0.1.jar:/shared/hive/lib/commons-cli-1.2.jar:/shared/hive/lib/commons-codec-1.3.jar:/shared/hive/lib/commons-collections-3.2.1.jar:/shared/hive/lib/commons-dbcp-1.4.jar:/shared/hive/lib/commons-lang-2.4.jar:/shared/hive/lib/commons-logging-1.0.4.jar:/shared/hive/lib/commons-logging-api-1.0.4.jar:/shared/hive/lib/commons-pool-1.5.4.jar:/shared/hive/lib/datanucleus-connectionpool-2.0.3.jar:/shared/hive/lib/datanucleus-core-2.0.3.jar:/shared/hive/lib/datanucleus-enhancer-2.0.3.jar:/shared/hive/lib/datanucleus-rdbms-2.0.3.jar:/shared/hive/lib/derby-10.4.2.0.jar:/shared/hive/lib/guava-r09.jar:/shared/hive/lib/hbase-0.92.0.jar:/shared/hive/lib/hbase-0.92.0-tests.jar:/shared/hive/lib/hive-builtins-0.9.0.jar:/shared/hive/lib/hive-cli-0.9.0.jar:/shared/hive/lib/hive-common-0.9.0.jar:/shared/hive/lib/hive-contrib-0.9.0.jar:/shared/hive/lib/hive_contrib.jar:/shared/hive/lib/hive-exec-0.9.0.jar:/shared/hive/lib/hive-hbase-handler-0.9.0.jar:/shared/hive/lib/hive-hwi-0.9.0.jar:/shared/hive/lib/hive-jdbc-0.9.0.jar:/shared/hive/lib/hive-metastore-0.9.0.jar:/shared/hive/lib/hive-pdk-0.9.0.jar:/shared/hive/lib/hive-serde-0.9.0.jar:/shared/hive/lib/hive-service-0.9.0.jar:/shared/hive/lib/hive-shims-0.9.0.jar:/shared/hive/lib/jackson-core-asl-1.8.8.jar:/shared/hive/lib/jackson-jaxrs-1.8.8.jar:/shared/hive/lib/jackson-mapper-asl-1.8.8.jar:/shared/hive/lib/jackson-xc-1.8.8.jar:/shared/hive/lib/JavaEWAH-0.3.2.jar:/shared/hive/lib/jdo2-api-2.3-ec.jar:/shared/hive/lib/jline-0.9.94. > > steve@mithril:/shared/hive/bin$ *ls -lrta /shared/hive/conf/hive-site.xml* > -rw-r--r-- 1 steve steve 4415 2012-11-25 23:02 > /shared/hive/conf/hive-site.xml > > > > 2012/11/30 Lauren Yang <lauren.y...@microsoft.com> > >> You can see if the classpath is being passed correctly to hadoop by >> putting in an echo statement around line 150 of the hive cli script where >> it passes the CLASSPATH variable to HADOOP_CLASSPATH.**** >> >> # pass classpath to hadoop**** >> >> export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${CLASSPATH}"**** >> >> ** ** >> >> You could also echo the classpath in the hadoop script (in your >> HADOOP_HOME\bin directory) to see if the classpath is being passed >> correctly to the time when the cli jar is invoked.**** >> >> ** ** >> >> As far as the logs location, if this is not set in your hive-site.xml, >> you can set it by passing in HIVE_OPTS when you invoke the command line. >> **** >> >> ** ** >> >> Like so:**** >> >> EXPORT HIVE_OPTS= -hiveconf hive.log.dir=$ HIVE_HOME\logs"**** >> >> Then run “hive”**** >> >> ** ** >> >> Or:**** >> >> Run “hive --hiveconf hive.log.dir=$ HIVE_HOME\logs”**** >> >> ** ** >> >> Thanks,**** >> >> Lauren**** >> >> ** ** >> >> ** ** >> >> *From:* Stephen Boesch [mailto:java...@gmail.com] >> *Sent:* Friday, November 30, 2012 12:16 AM >> *To:* user@hive.apache.org >> *Subject:* Re: hive-site.xml not found on classpath**** >> >> ** ** >> >> running 0.9.0 (you can see it from the classpath shown below);**** >> >> ** ** >> >> steve@mithril:/shared/cdh4$ echo $HIVE_CONF_DIR**** >> >> /shared/hive/conf**** >> >> steve@mithril:/shared/cdh4$ ls -l $HIVE_CONF_DIR**** >> >> total 152**** >> >> -rw-r--r-- 1 steve steve 46053 2011-12-13 00:36 hive-default.xml.template >> **** >> >> -rw-r--r-- 1 steve steve 1615 2012-11-13 23:37 hive-env.bullshit.sh**** >> >> -rw-r--r-- 1 steve steve 1671 2012-11-28 01:43 hive-env.sh**** >> >> -rw-r--r-- 1 steve steve 1593 2011-12-13 00:36 hive-env.sh.template**** >> >> -rw-r--r-- 1 steve steve 1637 2011-12-13 00:36 >> hive-exec-log4j.properties.template**** >> >> -rw-r--r-- 1 root root 2056 2012-11-28 01:38 hive-log4j.properties**** >> >> -rw-r--r-- 1 steve steve 2056 2012-03-25 12:49 >> hive-log4j.properties.template**** >> >> -rw-r--r-- 1 steve steve 4415 2012-11-25 23:02 hive-site.xml**** >> >> steve@mithril:/shared/cdh4$ echo $HIVE_HOME**** >> >> /shared/hive**** >> >> steve@mithril:/shared/cdh4$ echo $(which hive)**** >> >> /shared/hive/bin/hive**** >> >> ** ** >> >> also you can see the hive/conf is the first entry**** >> >> ** ** >> >> After adding the debug statement: **** >> >> ** ** >> >> classpath=*/shared/hive/conf:* >> /shared/hive/lib/antlr-runtime-3.0.1.jar:/shared/hive/lib/commons-cli-1.2.jar:/shared/hive/lib/commons-codec-1.3.jar:/shared/hive/lib/commons-collections-3.2.1.jar:/shared/hive/lib/commons-dbcp-1.4.jar:/shared/hive/lib/commons-lang-2.4.jar:/shared/hive/lib/commons-logging-1.0.4.jar:/shared/hive/lib/commons-logging-api-1.0.4.jar:/shared/hive/lib/commons-pool-1.5.4.jar:/shared/hive/lib/datanucleus-connectionpool-2.0.3.jar:/shared/hive/lib/datanucleus-core-2.0.3.jar:/shared/hive/lib/datanucleus-enhancer-2.0.3.jar:/shared/hive/lib/datanucleus-rdbms-2.0.3.jar:/shared/hive/lib/derby-10.4.2.0.jar:/shared/hive/lib/guava-r09.jar:/shared/hive/lib/hbase-0.92.0.jar:/shared/hive/lib/hbase-0.92.0-tests.jar:/shared/hive/lib/hive-builtins-0.9.0.jar:/shared/hive/lib/hive-cli-0.9.0.jar:/shared/hive/lib/hive-common-0.9.0.jar:/shared/hive/lib/hive-contrib-0.9.0.jar:/shared/hive/lib/hive_contrib.jar:/shared/hive/lib/hive-exec-0.9.0.jar:/shared/hive/lib/hive-hbase-handler-0.9.0.jar:/shared/hive/lib/hive-hwi-0.9.0.jar:/shared/hive/lib/hive-jdbc-0.9.0.jar:/shared/hive/lib/hive-metastore-0.9.0.jar:/shared/hive/lib/hive-pdk-0.9.0.jar:/shared/hive/lib/hive-serde-0.9.0.jar:/shared/hive/lib/hive-service-0.9.0.jar:/shared/hive/lib/hive-shims-0.9.0.jar:/shared/hive/lib/jackson-core-asl-1.8.8.jar:/shared/hive/lib/jackson-jaxrs-1.8.8.jar:/shared/hive/lib/jackson-mapper-asl-1.8.8.jar:/shared/hive/lib/jackson-xc-1.8.8.jar:/shared/hive/lib/JavaEWAH-0.3.2.jar:/shared/hive/lib/jdo2-api-2.3-ec.jar:/shared/hive/lib/jline-0.9.94.jar:/shared/hive/lib/json-20090211.jar:/shared/hive/lib/libfb303-0.7.0.jar:/shared/hive/lib/libfb303.jar:/shared/hive/lib/libthrift-0.7.0.jar:/shared/hive/lib/libthrift.jar:/shared/hive/lib/log4j-1.2.16.jar:/shared/hive/lib/mysql-connector-java-5.1.18-bin.jar:/shared/hive/lib/slf4j-api-1.6.1.jar:/shared/hive/lib/slf4j-log4j12-1.6.1.jar:/shared/hive/lib/stringtemplate-3.1-b1.jar:/shared/hive/lib/zookeeper-3.4.3.jar: >> **** >> >> ** ** >> >> ** ** >> >> But even so:**** >> >> - the log dir is still wrong (writing to /tmp/${user}/hive.log >> instead of $HIVE_HOME/logs)**** >> - the following message in the log file**** >> >> 2012-11-30 00:12:31,775 WARN conf.HiveConf >> (HiveConf.java:<clinit>(70)) - *hive-site.xml not found on CLASSPATH***** >> >> * ***** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> 2012/11/30 Bing Li <sarah.lib...@gmail.com>**** >> >> which version of hive do you use? >> >> Could you try to add the following debug line in bin/hive before hive >> real executes, and see the result? >> >> *echo "CLASSPATH=$CLASSPATH"* >> >> if [ "$TORUN" = "" ]; then >> echo "Service $SERVICE not found" >> echo "Available Services: $SERVICE_LIST" >> exit 7 >> else >> $TORUN "$@" >> fi >> >> The version I used is 0.9.0**** >> >> >> >> **** >> >> 2012/11/30 Stephen Boesch <java...@gmail.com>**** >> >> Yes i do mean the log is in the wrong location, since it was set to a >> persistent path in the $HIVE_CONF_DIR/lhive-log4j.properties.**** >> >> ** ** >> >> None of the files in that directory appear to be picked up properly: >> neither the hive-site.xml nor log4j.properties.**** >> >> ** ** >> >> I have put echo statements into the 'hive" and the hive-config.sh shell >> scripts and the echo statements prove that HIVE_CONF_DIR is set properly: >> /shared/hive/conf **** >> >> ** ** >> >> But even so the following problems occur:**** >> >> - the message "hive-site.xml is not found in the classpath"**** >> - none of the hive-site.xml values are taking properly**** >> - the log4j.properties in that same directory is not taking effect.*** >> * >> >> ** ** >> >> ** ** >> >> 2012/11/29 Bing Li <sarah.lib...@gmail.com>**** >> >> Hi, Stephen >> what did you mean the "wrong place under /tmp" in >> "I am seeing the following message in the logs (which are in the wrong >> place under /tmp..)" ? >> >> Did you mean that you set a different log dir but it didn't work? >> >> the log dir should be set in conf/hive-log4j.properties, >> conf/hive-exec-log4j.properties >> and you can try to reset HIVE_CONF_DIR in conf/hive-env.sh with ‘export" >> command. >> >> - Bing**** >> >> ** ** >> >> 2012/11/30 Stephen Boesch <java...@gmail.com>**** >> >> thought i mentioned in the posts those were already set and verified.. >> but yes in any case that's first thing looked at. >> >> steve@mithril:~$ echo $HIVE_CONF_DIR >> /shared/hive/conf >> steve@mithril:~$ echo $HIVE_HOME >> /shared/hive**** >> >> ** ** >> >> 2012/11/29 kulkarni.swar...@gmail.com <kulkarni.swar...@gmail.com>**** >> >> ** ** >> >> Have you tried setting HIVE_HOME and HIVE_CONF_DIR?**** >> >> ** ** >> >> On Thu, Nov 29, 2012 at 2:46 PM, Stephen Boesch <java...@gmail.com> >> wrote:**** >> >> Yes. **** >> >> ** ** >> >> 2012/11/29 Shreepadma Venugopalan <shreepa...@cloudera.com>**** >> >> Are you seeing this message when your bring up the standalone hive cli by >> running 'hive'?**** >> >> ** ** >> >> On Thu, Nov 29, 2012 at 12:56 AM, Stephen Boesch <java...@gmail.com> >> wrote:**** >> >> i am running under user steve. the latest log (where this shows up ) is >> /tmp/steve/hive.log**** >> >> ** ** >> >> 2012/11/29 Viral Bajaria <viral.baja...@gmail.com>**** >> >> You are seeing this error when you run the hive cli or in the tasktracker >> logs when you run a query ?**** >> >> ** ** >> >> On Thu, Nov 29, 2012 at 12:42 AM, Stephen Boesch <java...@gmail.com> >> wrote:**** >> >> ** ** >> >> I am seeing the following message in the logs (which are in the wrong >> place under /tmp..)**** >> >> ** ** >> >> hive-site.xml not found on classpath**** >> >> ** ** >> >> My hive-site.xml is under the standard location $HIVE_HOME/conf so this >> should not happen.**** >> >> ** ** >> >> Now some posts have talked that the HADOOP_CLASSPATH was mangled. Mine >> is not..**** >> >> ** ** >> >> So what is the underlying issue here?**** >> >> ** ** >> >> Thanks**** >> >> ** ** >> >> stephenb**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> >> >> **** >> >> ** ** >> >> -- >> Swarnim**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> >> ** ** >> > >