On Fri, Jan 12, 2018 at 4:13 AM, Eric Charles <e...@apache.org> wrote: >> Again, I don't see what is all this hoopla about fine grained control >> of dependency downloads. Spark solved this years ago for Spark >> applications. Don't reinvent the wheel. > > Init-containers are used today to download dependencies. I may be wrong and > may open another facet of the discussion, but I see init container usage in > a more generic way and not only restricted to dependencies download.
I'm not trying to discuss the general benefits of init containers as it pertains to the kubernetes framework. I'm sure they added those for a reason. I'm trying to discuss them in the restricted scope of the spark-on-k8s integration. And there, there is a single use for the single init container that the Spark code itself injects into the pod: downloading dependencies, which is something that spark-submit already does. There's an option to override that one init container image with another, where you can completely change its behavior. Given that there is no contract currently that explains how these images should behave, doing so is very, very risky and might break the application completely (e.g. because dependencies are now not being downloaded, or placed in the wrong location). An you can do the exact same thing by overriding the main Spark image itself. Just run the same code in your custom entry point before the Spark-provided entry point runs. Same results and caveats as above apply. So again, the specific init container used by spark-on-k8s, as far as I can see, seems to cause more problems than it solves. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
