Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and TaskSetManager and it does looked like tasks are directly sent to executors. Also would love to be corrected if mistaken as I have little knowledge about Spark internals and very new at scala.
On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote: > On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati < > nisrina.luthfiy...@gmail.com> wrote: > >> Hi all, >> I'm trying to understand how yarn-client mode works and found these two >> diagrams: >> >> >> >> >> In the first diagram, it looks like the driver running in client directly >> communicates with executors to issue application commands, while in the >> second diagram it looks like application commands is sent to application >> master first and then forwarded to executors. >> > > My limited understanding tells me that regardless of deploy mode (local, > standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends > TaskSets to executors once they're launched. YARN and Mesos are only used > until they offer resources (CPU and memory) and once executors start, these > cluster managers are not engaged in the communication (driver and executors > communicate using RPC over netty since 1.6-SNAPSHOT or akka before). > > I'd love being corrected if mistaken. Thanks. > > Jacek > -- Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010 http://www.facebook.com/nisrina.luthfiyati http://id.linkedin.com/in/nisrina
