Re: Where does the Driver run?

8, 2019 10:10 AM > *To:* Jianneng Li > *Cc:* user@spark.apache.org; ak...@hacked.work; andrew.m...@gmail.com; > and...@actionml.com > > *Subject:* Re: Where does the Driver run? > > Thanks for the pointers. We’ll investigate. > > We have been told that the “Driver” is run in

Re: Where does the Driver run?

s and call SparkContext accordingly. Best, Jianneng From: Pat Ferrel Sent: Thursday, March 28, 2019 10:10 AM To: Jianneng Li Cc: user@spark.apache.org; ak...@hacked.work; andrew.m...@gmail.com; and...@actionml.com Subject: Re: Where does the Driver ru

Re: Where does the Driver run?

...@hacked.work Subject: Re: Where does the Driver run? Hi Pat, The driver runs in the same JVM as SparkContext. You didn't go into detail about how you "launch" the job (i.e. how the SparkContext is created), so it's hard for me to guess where the driver is. For reference, we've had success

Re: Where does the Driver run?

Hi, I have explained this in my following Linkedlin article "The Operational Advantages of Spark as a Distributed Processing Framework " An extract *2) YARN Deployment Modes* The term D*eployment mode of

Re: Where does the Driver run?

w Melo Cc: user , Akhil Das Subject: Re: Where does the Driver run? I’m beginning to agree with you and find it rather surprising that this is mentioned nowhere explicitly (maybe I missed?). It is possible to serialize code to be executed in executors to various nodes. It also seem

Re: Where does the Driver run?

:07 AM To: Pat Ferrel Cc: Akhil Das , user Subject: Re: Where does the Driver run? Hi Pat, Indeed, I don't think that it's possible to use cluster mode w/o spark-submit. All the docs I see appear to always describe needing to use spark-submit for cluster mode -- it's not even compatible

Re: Where does the Driver run?

bed unless you use > spark-submit (and I’m only guessing at that). > > Further; if we don’t use spark-submit we can’t use deployMode = cluster ??? > > > From: Akhil Das > Reply: Akhil Das > Date: March 24, 2019 at 7:45:07 PM > To: Pat Ferrel > Cc: user > Sub

Re: Where does the Driver run?

only guessing at that). Further; if we don’t use spark-submit we can’t use deployMode = cluster ??? From: Akhil Das Reply: Akhil Das Date: March 24, 2019 at 7:45:07 PM To: Pat Ferrel Cc: user Subject: Re: Where does the Driver run? There's also a driver ui (usually available on port 4040

Re: Where does the Driver run?

e: March 23, 2019 at 9:26:50 PM > To: Pat Ferrel > Cc: user > Subject: Re: Where does the Driver run? > > If you are starting your "my-app" on your local machine, that's where the > driver is running. > > [image: image.png] > > Hope this helps. > <h

Re: Where does the Driver run?

Driver, and the Master > on 2 Workers. > > > > > From: Andrew Melo > Reply: Andrew Melo > Date: March 24, 2019 at 12:46:35 PM > To: Pat Ferrel > Cc: Akhil Das , user > > Subject: Re: Where does the Driver run? > > Hi Pat, > > On Sun, Mar 24, 2019

Re: Where does the Driver run?

60g BTW I would expect this to create one Executor, one Driver, and the Master on 2 Workers. From: Andrew Melo Reply: Andrew Melo Date: March 24, 2019 at 12:46:35 PM To: Pat Ferrel Cc: Akhil Das , user Subject: Re: Where does the Driver run? Hi Pat, On Sun, Mar 24, 2019 at 1:03 PM

Re: Where does the Driver run?

60g From: Andrew Melo Reply: Andrew Melo Date: March 24, 2019 at 12:46:35 PM To: Pat Ferrel Cc: Akhil Das , user Subject: Re: Where does the Driver run? Hi Pat, On Sun, Mar 24, 2019 at 1:03 PM Pat Ferrel wrote: > Thanks, I have seen this many times in my research. Paraphrasi

Re: Where does the Driver run?

wonder if > the Driver only runs in the cluster if we use spark-submit > Where/how are you starting "./sbin/start-master.sh"? Cheers Andrew > > > > From: Akhil Das > Reply: Akhil Das > Date: March 23, 2019 at 9:26:50 PM > To: Pat Ferrel &g

Re: Where does the Driver run?

: Akhil Das Date: March 23, 2019 at 9:26:50 PM To: Pat Ferrel Cc: user Subject: Re: Where does the Driver run? If you are starting your "my-app" on your local machine, that's where the driver is running. [image: image.png] Hope this helps. <https://spark.apache.org/docs/l

Re: Where does the Driver run?

If you are starting your "my-app" on your local machine, that's where the driver is running. [image: image.png] Hope this helps. On Sun, Mar 24, 2019 at 4:13 AM Pat Ferrel wrote: > I have researched this for a significant amount of

Where does the Driver run?

I have researched this for a significant amount of time and find answers that seem to be for a slightly different question than mine. The Spark 2.3.3 cluster is running fine. I see the GUI on “ http://master-address:8080;, there are 2 idle workers, as configured. I have a Scala application that