You'd need the jar file (holding class definitions etc.) to do the
deserialisation on the executor.
On 18 June 2015 at 03:48, Shiyao Ma i...@introo.me wrote:
Hi,
Looking from my executor logs, the submitted application jar is
transmitted to each executors?
Why does spark do the above? To my
Hi,
Looking from my executor logs, the submitted application jar is
transmitted to each executors?
Why does spark do the above? To my understanding, the tasks to be run
are already serialized with TaskDescription.
Regards.
-
TaskDescription only serialize the jar path not the jar content. Multiple
tasks can run on the same executor. Executor will check whether the jar has
been fetched when each time task is launched. If so, it won't fetch it
again.
Only serialize the jar path can prevent serialize jar multiple times
