Hi Linlin,
have you got the solution for this issue, if yes then what are the thing
need to make correct,because I am also getting same error,when submitting
spark job in cluster mode getting error as under -
2015-04-14 18:16:43 DEBUG Transaction - Transaction rolled back in 0 ms
2015-04-14
have HiveContext API to talk to Hive metastore,
and vice versa. Thanks.
Date: Thu, 5 Feb 2015 16:59:12 -0800
From: lian.cs@gmail.com
To: linlin200...@gmail.com; huaiyin@gmail.com
CC: user@spark.apache.org
Subject: Re: Spark sql failed in yarn-cluster mode when connecting to
non-default
Hi Jenny,
You may try to use |--files $SPARK_HOME/conf/hive-site.xml
--driver-class-path hive-site.xml| when submitting your application. The
problem is that when running in cluster mode, the driver is actually
running in a random container directory on a random executor node. By
using
(javax.jdo.option.ConnectionPassword,xxx)
hiveContext.setConf(javax.jdo.option.ConnectionURL,jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true)
From: huaiyin@gmail.com
Date: Wed, 13 Aug 2014 16:56:13 -0400
Subject: Re: Spark sql failed in yarn-cluster mode when
with -Phadoop-provided, and other common libraries that
are required.
From: alee...@hotmail.com
To: user@spark.apache.org
CC: lian.cs@gmail.com; linlin200...@gmail.com; huaiyin@gmail.com
Subject: RE: Spark sql failed in yarn-cluster mode when connecting to
non-default hive database
Date
I think the problem is that when you are using yarn-cluster mode, because
the Spark driver runs inside the application master, the hive-conf is not
accessible by the driver. Can you try to set those confs by using
hiveContext.set(...)? Or, maybe you can copy hive-site.xml to spark/conf in
the node
Hi Jenny,
Have you copied hive-site.xml to spark/conf directory? If not, can you put
it in conf/ and try again?
Thanks,
Yin
On Mon, Aug 11, 2014 at 8:57 PM, Jenny Zhao linlin200...@gmail.com wrote:
Thanks Yin!
here is my hive-site.xml, which I copied from $HIVE_HOME/conf, didn't
Hi Yin,
hive-site.xml was copied to spark/conf and the same as the one under
$HIVE_HOME/conf.
through hive cli, I don't see any problem. but for spark on yarn-cluster
mode, I am not able to switch to a database other than the default one, for
Yarn-client mode, it works fine.
Thanks!
Jenny
On
Since you were using hql(...), it’s probably not related to JDBC driver.
But I failed to reproduce this issue locally with a single node pseudo
distributed YARN cluster. Would you mind to elaborate more about steps to
reproduce this bug? Thanks
On Sun, Aug 10, 2014 at 9:36 PM, Cheng Lian
Hi Jenny,
How's your metastore configured for both Hive and Spark SQL? Which
metastore mode are you using (based on
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
)?
Thanks,
Yin
On Mon, Aug 11, 2014 at 6:15 PM, Jenny Zhao linlin200...@gmail.com wrote:
you can
Thanks Yin!
here is my hive-site.xml, which I copied from $HIVE_HOME/conf, didn't
experience problem connecting to the metastore through hive. which uses DB2
as metastore database.
?xml version=1.0?
?xml-stylesheet type=text/xsl href=configuration.xsl?
!--
Licensed to the Apache Software
Hi Jenny, does this issue only happen when running Spark SQL with YARN in
your environment?
On Sat, Aug 9, 2014 at 3:56 AM, Jenny Zhao linlin200...@gmail.com wrote:
Hi,
I am able to run my hql query on yarn cluster mode when connecting to the
default hive metastore defined in
12 matches
Mail list logo
