TaskDescription only serialize the jar path not the jar content. Multiple tasks can run on the same executor. Executor will check whether the jar has been fetched when each time task is launched. If so, it won't fetch it again. Only serialize the jar path can prevent serialize jar multiple times which is inefficient.
On Thu, Jun 18, 2015 at 10:48 AM, Shiyao Ma <i...@introo.me> wrote: > Hi, > > Looking from my executor logs, the submitted application jar is > transmitted to each executors? > > Why does spark do the above? To my understanding, the tasks to be run > are already serialized with TaskDescription. > > > Regards. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Best Regards Jeff Zhang
