We have two copies of hive running in order to support multiple versions of hive with a single version of Spark. You are see log messages for the version that we use for execution (it just creates a temporary derby metastore).
On Mon, Nov 9, 2015 at 3:32 PM, mayurladwa <mayur.la...@blackrock.com> wrote: > Hello, this question is around the hive thrift server that comes packaged > with spark 1.5.1, I am trying to change the default metastore from derby. > > From googling I see the more commonly documented alternatives to derby are > MySQL, but as it exposes a JDBC interface I want to try and get this > working > with HSQL (2.0). > > I'm overriding the following in the hive-site.xml: > > /<property> > <name>javax.jdo.option.ConnectionURL</name> > <value>jdbc:hsqldb:hsql://myhost:58090/default</value> > <description>JDBC connect string for a JDBC metastore</description> > </property> > > <property> > <name>javax.jdo.option.ConnectionDriverName</name> > <value>org.hsqldb.jdbc.JDBCDriver</value> > <description>Driver class name for a JDBC metastore</description> > </property> > > <property> > <name>javax.jdo.option.ConnectionUserName</name> > <value>user</value> > <description>username to use against metastore database</description> > </property> > > <property> > <name>javax.jdo.option.ConnectionPassword</name> > <value>pwd</value> > <description>password to use against metastore database</description> > </property> > / > > What's really strange is that I see some hive tables created in my HSQL > database when my spark hive thrift server is running, but when I do a query > I see it switches back to derby! I get logs like this: > > /15/11/09 11:59:16 DEBUG ObjectStore: *Overriding > javax.jdo.option.ConnectionURL value null from jpox.properties with > jdbc:hsqldb:hsql://myhost:58090/default*/ > > And then later I see this: > > /15/11/09 11:59:18 DEBUG ObjectStore: *Overriding > javax.jdo.option.ConnectionURL value null from jpox.properties with > > jdbc:derby:;databaseName=/tmp/spark-acb48194-09a7-4beb-b5fc-*ffc0216449c8/metastore;create=true/ > ... > /15/11/09 11:59:18 DEBUG Transaction: Transaction committed in 1 ms > 15/11/09 11:59:18 INFO MetaStoreDirectSql: Using direct SQL, *underlying DB > is DERBY* > 15/11/09 11:59:18 DEBUG ObjectStore: RawStore: > org.apache.hadoop.hive.metastore.ObjectStore@7ecab68e, with > PersistenceManager: org.datanucleus.api.jdo.JDOPersistenceManager@61e68dab > created in the thread with id: 1 > 15/11/09 11:59:18 INFO ObjectStore: Initialized ObjectStore/ > > So not totally sure how this is getting switched back to derby, or why it > thinks later on that the jpox.properties I am overriding in the > hive-site.xml area is suddenly null? > > Any help would be much appreciated :) > > Many thanks > > Mayur > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Overriding-Derby-in-hive-site-xml-giving-strange-results-tp25333.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
