Re: spark 1.6.0 connect to hive metastore

can someone provide the correct settings for spark 1.6.1 to work with cdh 5 (hive 1.1.0)? in particular the settings for: spark.sql.hive.version spark.sql.hive.metastore.jars also it would be helpful to know if your spark jar includes hadoop dependencies or not. i realize it works (or at least

Re: spark 1.6.0 connect to hive metastore

I had similar issue with CDH 5.5.3. Not only with Spark 1.6 but with beeline as well. I resolved it via installation & running hiveserver2 role instance at the same server wher metastore is. On Tue, Feb 9, 2016 at 10:58 PM, Koert Kuipers

Re: spark 1.6.0 connect to hive metastore

hive 1.6.0 in embed mode doesn't connect to metastore -- https://issues.apache.org/jira/browse/SPARK-9686 https://forums.databricks.com/questions/6512/spark-160-not-able-to-connect-to-hive-metastore.html On Wed, Mar 9, 2016 at 10:48 AM, Suniti Singh wrote: > Hi, > > I

Re: spark 1.6.0 connect to hive metastore

Hi, I am able to reproduce this error only when using spark 1.6.0 and hive 1.6.0. The hive-site.xml is in the classpath but somehow spark rejects the classpath search for hive-site.xml and start using the default metastore Derby. 16/03/09 10:37:52 INFO MetaStoreDirectSql: Using direct SQL,

Re: spark 1.6.0 connect to hive metastore

Hi, We're having a similar issue. We have a standalone cluster running 1.5.2 with Hive working fine having dropped hive-site.xml into the conf folder. We've just updated to 1.6.0, using the same configuration. Now when starting a spark-shell we get the following: java.lang.RuntimeException:

spark 1.6.0 connect to hive metastore

has anyone successfully connected to hive metastore using spark 1.6.0? i am having no luck. worked fine with spark 1.5.1 for me. i am on cdh 5.5 and launching spark with yarn. this is what i see in logs: 16/02/09 14:49:12 INFO hive.metastore: Trying to connect to metastore with URI

Re: spark 1.6.0 connect to hive metastore

hey thanks. hive-site is on classpath in conf directory i currently got it to work by changing this hive setting in hive-site.xml: hive.metastore.schema.verification=true to hive.metastore.schema.verification=false this feels like a hack, because schema verification is a good thing i would

Re: spark 1.6.0 connect to hive metastore

Hi Koert, As far as I can see you are using derby: Using direct SQL, underlying DB is DERBY not mysql, which is used for the metastore. That means, spark couldn't find hive-site.xml on your classpath. Can you check that, please? Thanks, Alex. On Tue, Feb 9, 2016 at 8:58 PM, Koert Kuipers

Re: spark 1.6.0 connect to hive metastore

I'm using spark 1.6.0, hive 1.2.1 and there is just one property in the hive-site.xml hive.metastore.uris Works for me. Can you check in the logs, that when the HiveContext is created it connects to the correct uri and doesn't use derby. Cheers, Alex. On Tue, Feb 9, 2016 at 9:39 PM, Koert

Re: spark 1.6.0 connect to hive metastore

I got the same problem when I added the Phoenix plugin jar in the driver and executor extra classpaths. Do you have those set too? > On Feb 9, 2016, at 1:12 PM, Koert Kuipers wrote: > > yes its not using derby i think: i can see the tables in my actual hive > metastore. >

Re: spark 1.6.0 connect to hive metastore

yes its not using derby i think: i can see the tables in my actual hive metastore. i was using a symlink to /etc/hive/conf/hive-site.xml for my hive-site.xml which has a lot more stuff than just hive.metastore.uris let me try your approach On Tue, Feb 9, 2016 at 3:57 PM, Alexandr Dzhagriev

Re: spark 1.6.0 connect to hive metastore

Hi, I am using by telling Spark about hive version we are using. This is done by setting following properties spark.sql.hive.version spark.sql.hive.metastore.jars Thanks On Wed, Feb 10, 2016 at 7:39 AM, Koert Kuipers wrote: > hey thanks. hive-site is on classpath in conf

Re: spark 1.6.0 connect to hive metastore

i do not have phoenix, but i wonder if its something related. will check my classpaths On Tue, Feb 9, 2016 at 5:00 PM, Benjamin Kim wrote: > I got the same problem when I added the Phoenix plugin jar in the driver > and executor extra classpaths. Do you have those set too? >