Re: Why does driver transfer application jar to executors?
You'd need the jar file (holding class definitions etc.) to do the deserialisation on the executor. On 18 June 2015 at 03:48, Shiyao Ma i...@introo.me wrote: Hi, Looking from my executor logs, the submitted application jar is transmitted to each executors? Why does spark do the above? To my understanding, the tasks to be run are already serialized with TaskDescription. Regards. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Why does driver transfer application jar to executors?
Hi, Looking from my executor logs, the submitted application jar is transmitted to each executors? Why does spark do the above? To my understanding, the tasks to be run are already serialized with TaskDescription. Regards. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Why does driver transfer application jar to executors?
TaskDescription only serialize the jar path not the jar content. Multiple tasks can run on the same executor. Executor will check whether the jar has been fetched when each time task is launched. If so, it won't fetch it again. Only serialize the jar path can prevent serialize jar multiple times which is inefficient. On Thu, Jun 18, 2015 at 10:48 AM, Shiyao Ma i...@introo.me wrote: Hi, Looking from my executor logs, the submitted application jar is transmitted to each executors? Why does spark do the above? To my understanding, the tasks to be run are already serialized with TaskDescription. Regards. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Best Regards Jeff Zhang
