Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
TaskSetManager and it does looked like tasks are directly sent to
executors. Also would love to be corrected if mistaken as I have little
knowledge about Spark internals and very new at scala.
On Tue, Dec 1, 2015 at 1:16 AM,
Hi,
That's my understanding, too. Just spent an entire morning today to check
it out and would be surprised to hear otherwise.
Pozdrawiam,
Jacek
--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark
Checkout the Sameer Farooqui video on youtube for spark internals
(https://www.youtube.com/watch?v=7ooZ4S7Ay6Y=PLIxzgeMkSrQ-2Uizm4l0HjNSSy2NxgqjX)
Starting at 2:15:00, he describes YARN mode.
btw, highly recommend the entire video. Very detailed and concise.
--
Ali
On Dec 7, 2015, at 8:38
On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
nisrina.luthfiy...@gmail.com> wrote:
> Hi all,
> I'm trying to understand how yarn-client mode works and found these two
> diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly
> communicates with
Hi,
In general YARN is used as the resource scheduler regardless of the execution
engine whether it is MapReduce or Spark.
Yarn will create a resource container for the submitted job (that is the Spark
client) and will execute it in the default engine (in this case Spark). There
will be
Hi all,
I'm trying to understand how yarn-client mode works and found these two
diagrams:
In the first diagram, it looks like the driver running in client directly
communicates with executors to issue application commands, while in the
second diagram it looks like application commands is sent
Hi Mich, thank you for the answer. Regarding the diagrams, I'm specifically
referring to the direct line between spark yarn client to spark executor in
the first diagram which implies direct communication to executor when
issuing application commands. And the 'Application commands' & 'Issue
