Re: SPark - YARN Cluster Mode

Hi Thanks a lot, i used property file to resolve the issue. I think documentation should mention it though. On Tue, 28 Feb 2017 at 5:05 am, Marcelo Vanzin wrote: > > none of my Config settings > > Is it none of the configs or just the queue? You can't set the YARN > queue

Re: SPark - YARN Cluster Mode

> none of my Config settings Is it none of the configs or just the queue? You can't set the YARN queue in cluster mode through code, it has to be set in the command line. It's a chicken & egg problem (in cluster mode, the YARN app is created before your code runs). --property-file works the

Re: SPark - YARN Cluster Mode

Also, I wanted to add if I specify the conf in the command line, it seems to be working. For example, if I use spark-submit --master yarn --deploy-mode cluster --conf spark.yarn.queue=root.Application ayan_test.py 10 Then it is going to correct queue. Any help would be great Best Ayan On

SPark - YARN Cluster Mode

Hi I am facing an issue with Cluster Mode, with pyspark Here is my code: conf = SparkConf() conf.setAppName("Spark Ingestion") conf.set("spark.yarn.queue","root.Applications") conf.set("spark.executor.instances","50")

Re: Error trying to connect to Hive from Spark (Yarn-Cluster Mode)

6 AM > *To:* Gangadhar, Anupama (623) > *Cc:* user @spark > *Subject:* Re: Error trying to connect to Hive from Spark (Yarn-Cluster > Mode) > > > > Is your Hive Thrift Server up and running on port > jdbc:hive2://10001? > > > > Do the following >

RE: Error trying to connect to Hive from Spark (Yarn-Cluster Mode)

ing to connect to Hive from Spark (Yarn-Cluster Mode) Is your Hive Thrift Server up and running on port jdbc:hive2://10001? Do the following netstat -alnp |grep 10001 and see whether it is actually running HTH Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view

RE: Error trying to connect to Hive from Spark (Yarn-Cluster Mode)

? In the cluster, transport mode is http and ssl is disabled. Thanks Anupama From: Deepak Sharma [mailto:deepakmc...@gmail.com] Sent: Saturday, September 17, 2016 8:35 AM To: Gangadhar, Anupama (623) Cc: spark users Subject: Re: Error trying to connect to Hive from Spark (Yarn-Cluster Mode) Hi

Re: Error trying to connect to Hive from Spark (Yarn-Cluster Mode)

Hi Anupama To me it looks like issue with the SPN with which you are trying to connect to hive2 , i.e. hive@hostname. Are you able to connect to hive from spark-shell? Try getting the tkt using any other user keytab but not hadoop services keytab and then try running the spark submit. Thanks

Re: Error trying to connect to Hive from Spark (Yarn-Cluster Mode)

Is your Hive Thrift Server up and running on port jdbc:hive2://10001? Do the following netstat -alnp |grep 10001 and see whether it is actually running HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Error trying to connect to Hive from Spark (Yarn-Cluster Mode)

Hi, I am trying to connect to Hive from Spark application in Kerborized cluster and get the following exception. Spark version is 1.4.1 and Hive is 1.2.1. Outside of spark the connection goes through fine. Am I missing any configuration parameters? ava.sql.SQLException: Could not open