use yarn :) "spark-submit --master yarn"
On Sun, Jan 15, 2017 at 7:55 PM, Darren Govoni <dar...@ontrenet.com> wrote: > So what was the answer? > > > > Sent from my Verizon, Samsung Galaxy smartphone > > -------- Original message -------- > From: Andrew Holway <andrew.hol...@otternetworks.de> > Date: 1/15/17 11:37 AM (GMT-05:00) > To: Marco Mistroni <mmistr...@gmail.com> > Cc: Neil Jonkers <neilod...@gmail.com>, User <user@spark.apache.org> > Subject: Re: Running Spark on EMR > > Darn. I didn't respond to the list. Sorry. > > > > On Sun, Jan 15, 2017 at 5:29 PM, Marco Mistroni <mmistr...@gmail.com> > wrote: > >> thanks Neil. I followed original suggestion from Andrw and everything is >> working fine now >> kr >> >> On Sun, Jan 15, 2017 at 4:27 PM, Neil Jonkers <neilod...@gmail.com> >> wrote: >> >>> Hello, >>> >>> Can you drop the url: >>> >>> spark://master:7077 >>> >>> The url is used when running Spark in standalone mode. >>> >>> Regards >>> >>> >>> -------- Original message -------- >>> From: Marco Mistroni >>> Date:15/01/2017 16:34 (GMT+02:00) >>> To: User >>> Subject: Running Spark on EMR >>> >>> hi all >>> could anyone assist here? >>> i am trying to run spark 2.0.0 on an EMR cluster,but i am having issues >>> connecting to the master node >>> So, below is a snippet of what i am doing >>> >>> >>> sc = SparkSession.builder.master(sparkHost).appName("DataProcess" >>> ).getOrCreate() >>> >>> sparkHost is passed as input parameter. that was thought so that i can >>> run the script locally >>> on my spark local instance as well as submitting scripts on any cluster >>> i want >>> >>> >>> Now i have >>> 1 - setup a cluster on EMR. >>> 2 - connected to masternode >>> 3 - launch the command spark-submit myscripts.py spark://master:7077 >>> >>> But that results in an connection refused exception >>> Then i have tried to remove the .master call above and launch the script >>> with the following command >>> >>> spark-submit --master spark://master:7077 myscript.py but still i am >>> getting >>> connectionREfused exception >>> >>> >>> I am using Spark 2.0.0 , could anyone advise on how shall i build the >>> spark session and how can i submit a pythjon script to the cluster? >>> >>> kr >>> marco >>> >> >> > > > -- > Otter Networks UG > http://otternetworks.de > Gotenstraße 17 > 10829 Berlin > -- Otter Networks UG http://otternetworks.de Gotenstraße 17 10829 Berlin