Looking at this again, I think one could just use set `spark.yarn.dist.archives` in the conf passed to the job. If that works, please disregard :)
On Mon, Sep 17, 2018 at 1:09 PM Ben Laird <br.la...@gmail.com> wrote: > The current SparkSubmitHook doesn't appear to support the --archives flag. > From the Spark docs: > > "spark.yarn.dist.archives (none): Comma separated list of archives to be > extracted into the working directory of each executor." > https://spark.apache.org/docs/latest/running-on-yarn.html > > This is necessary for deploying zipped virtualenvs or other packages > across the cluster. For now, I'll have to maintain my own copy of this Hook > but will contribute this back if others aren't planning on it. > > Curious if there is context on why this isn't included? > > Ben >
