the pictures to illustrate it http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_running_spark_on_yarn.html
On Tue, Feb 9, 2016 at 10:18 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote: > In yarn-client mode, the driver is separate from the AM. The AM is created > in YARN, and YARN controls where it goes (though you can somewhat control > it using YARN node labels--I just learned earlier today in a different > thread on this list that this can be controlled by > spark.yarn.am.labelExpression). Then what I understand is that the driver > talks to the AM in order to request additional YARN containers in which to > run executors. > > In yarn-cluster mode, the SparkSubmit process outside of the cluster > creates the AM in YARN, and then what I understand is that the AM *becomes* > the driver (by invoking the driver's main method), and then it requests the > executor containers. > > So yes, one difference between yarn-client and yarn-cluster mode is that > in yarn-client mode the driver and AM are separate, whereas they are the > same in yarn-cluster. > > ~ Jonathan > > On Tue, Feb 9, 2016 at 9:57 PM praveen S <mylogi...@gmail.com> wrote: > >> Can you explain what happens in yarn client mode? >> >> Regards, >> Praveen >> On 10 Feb 2016 10:55, "ayan guha" <guha.a...@gmail.com> wrote: >> >>> It depends on yarn-cluster and yarn-client mode. >>> >>> On Wed, Feb 10, 2016 at 3:42 PM, praveen S <mylogi...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have 2 questions when running the spark jobs on yarn in client mode : >>>> >>>> 1) Where is the AM(application master) created : >>>> >>>> A) is it created on the client where the job was submitted? i.e driver >>>> and AM on the same client? >>>> Or >>>> B) yarn decides where the the AM should be created? >>>> >>>> 2) Driver and AM run in different processes : is my assumption correct? >>>> >>>> Regards, >>>> Praveen >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Ayan Guha >>> >>
