Checkout the Sameer Farooqui video on youtube for spark internals (https://www.youtube.com/watch?v=7ooZ4S7Ay6Y&list=PLIxzgeMkSrQ-2Uizm4l0HjNSSy2NxgqjX) Starting at 2:15:00, he describes YARN mode.
btw, highly recommend the entire video. Very detailed and concise. -- Ali On Dec 7, 2015, at 8:38 AM, Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > That's my understanding, too. Just spent an entire morning today to check it > out and would be surprised to hear otherwise. > > Pozdrawiam, > Jacek > > -- > Jacek Laskowski | https://medium.com/@jaceklaskowski/ | > http://blog.jaceklaskowski.pl > Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ > Follow me at https://twitter.com/jaceklaskowski > Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski > > On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati > <nisrina.luthfiy...@gmail.com> wrote: > Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and > TaskSetManager and it does looked like tasks are directly sent to executors. > Also would love to be corrected if mistaken as I have little knowledge about > Spark internals and very new at scala. > > On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote: > On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati > <nisrina.luthfiy...@gmail.com> wrote: > Hi all, > I'm trying to understand how yarn-client mode works and found these two > diagrams: > > > > > In the first diagram, it looks like the driver running in client directly > communicates with executors to issue application commands, while in the > second diagram it looks like application commands is sent to application > master first and then forwarded to executors. > > My limited understanding tells me that regardless of deploy mode (local, > standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends TaskSets > to executors once they're launched. YARN and Mesos are only used until they > offer resources (CPU and memory) and once executors start, these cluster > managers are not engaged in the communication (driver and executors > communicate using RPC over netty since 1.6-SNAPSHOT or akka before). > > I'd love being corrected if mistaken. Thanks. > > Jacek > > > > -- > Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010 > http://www.facebook.com/nisrina.luthfiyati > http://id.linkedin.com/in/nisrina > >
