In yarn-client mode, the driver is separate from the AM. The AM is created in YARN, and YARN controls where it goes (though you can somewhat control it using YARN node labels--I just learned earlier today in a different thread on this list that this can be controlled by spark.yarn.am.labelExpression). Then what I understand is that the driver talks to the AM in order to request additional YARN containers in which to run executors.
In yarn-cluster mode, the SparkSubmit process outside of the cluster creates the AM in YARN, and then what I understand is that the AM *becomes* the driver (by invoking the driver's main method), and then it requests the executor containers. So yes, one difference between yarn-client and yarn-cluster mode is that in yarn-client mode the driver and AM are separate, whereas they are the same in yarn-cluster. ~ Jonathan On Tue, Feb 9, 2016 at 9:57 PM praveen S <mylogi...@gmail.com> wrote: > Can you explain what happens in yarn client mode? > > Regards, > Praveen > On 10 Feb 2016 10:55, "ayan guha" <guha.a...@gmail.com> wrote: > >> It depends on yarn-cluster and yarn-client mode. >> >> On Wed, Feb 10, 2016 at 3:42 PM, praveen S <mylogi...@gmail.com> wrote: >> >>> Hi, >>> >>> I have 2 questions when running the spark jobs on yarn in client mode : >>> >>> 1) Where is the AM(application master) created : >>> >>> A) is it created on the client where the job was submitted? i.e driver >>> and AM on the same client? >>> Or >>> B) yarn decides where the the AM should be created? >>> >>> 2) Driver and AM run in different processes : is my assumption correct? >>> >>> Regards, >>> Praveen >>> >> >> >> >> -- >> Best Regards, >> Ayan Guha >> >
