Got it, then you can have an extracted Spark directory on each host on the same location, and don’t specify SPARK_EXECUTOR_URI. Instead, set spark.mesos.executor.home to that directory.
This should effectively do what you want, which avoids extracting and fetching and just executed the command. Tim On Fri, Jul 6, 2018 at 5:57 PM Tien Dat <tphan....@gmail.com> wrote: > Thank you for your answer. > > The think it I actually pointed to a local binary file. And Mesos locally > copied the binary file to a specific folder in /var/lib/mesos/... and > extract it to every time it launched an Spark executor. With the fetch > cache, the copy time is reduced, but the reduction is not much since the > file is stored at local any way. > The process that takes more time is the extraction. > Finally, since Mesos make a new folder for extracting the Spark binary each > time a new Spark job runs, the disk usage increases gradually. > > Therefore, our expectation is to have Spark running on Mesos without this > binary extraction, as well as without storing the same binary every time > new > Spark job runs. > > Does that make sense to you? And do you have any idea how to deal with > this? > > Best > > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
