Hi, That's my understanding, too. Just spent an entire morning today to check it out and would be surprised to hear otherwise.
Pozdrawiam, Jacek -- Jacek Laskowski | https://medium.com/@jaceklaskowski/ | http://blog.jaceklaskowski.pl Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ Follow me at https://twitter.com/jaceklaskowski Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati < nisrina.luthfiy...@gmail.com> wrote: > Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and > TaskSetManager and it does looked like tasks are directly sent to > executors. Also would love to be corrected if mistaken as I have little > knowledge about Spark internals and very new at scala. > > On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote: > >> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati < >> nisrina.luthfiy...@gmail.com> wrote: >> >>> Hi all, >>> I'm trying to understand how yarn-client mode works and found these two >>> diagrams: >>> >>> >>> >>> >>> In the first diagram, it looks like the driver running in client >>> directly communicates with executors to issue application commands, while >>> in the second diagram it looks like application commands is sent to >>> application master first and then forwarded to executors. >>> >> >> My limited understanding tells me that regardless of deploy mode (local, >> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends >> TaskSets to executors once they're launched. YARN and Mesos are only used >> until they offer resources (CPU and memory) and once executors start, these >> cluster managers are not engaged in the communication (driver and executors >> communicate using RPC over netty since 1.6-SNAPSHOT or akka before). >> >> I'd love being corrected if mistaken. Thanks. >> >> Jacek >> > > > > -- > Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010 > http://www.facebook.com/nisrina.luthfiyati > http://id.linkedin.com/in/nisrina > >
