Hi,

 

In general YARN is used as the resource scheduler regardless of the execution 
engine whether it is MapReduce or Spark.

 

Yarn will create a resource container for the submitted job (that is the Spark 
client) and will execute it in the default engine (in this case Spark). There 
will be a job scheduler and one or more Spark Executors depending on the 
cluster. So as far as I can see both diagrams are correct,

 

HTH

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

 
<http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf>
 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", 
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one 
out shortly

 

 <http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by Peridale Technology Ltd, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any 
responsibility.

 

From: Nisrina Luthfiyati [mailto:nisrina.luthfiy...@gmail.com] 
Sent: 27 November 2015 11:12
To: user@spark.apache.org
Subject: In yarn-client mode, is it the driver or application master that issue 
commands to executors?

 

Hi all,

I'm trying to understand how yarn-client mode works and found these two 
diagrams:


  
<http://image.slidesharecdn.com/th-1150a-hall1-feng-v2-140617142634-phpapp01/95/sparkonyarn-empower-spark-applications-on-hadoop-cluster-10-638.jpg?cb=1403015417>
 

  <http://blog.cloudera.com/wp-content/uploads/2014/05/spark-yarn-f22.png> 

In the first diagram, it looks like the driver running in client directly 
communicates with executors to issue application commands, while in the second 
diagram it looks like application commands is sent to application master first 
and then forwarded to executors. 

Would anyone knows which case is true or is there any other interpretation to 
these diagrams?

Thanks!

Nisrina

Reply via email to