You don't need to create the init-container. It's an implementation detail. If you provide a remote uri, and specify spark.kubernetes.container.image=<spark-image>, Spark *internally* will add the init container to the pod spec for you. *If *for some reason, you want to customize the init container image, you can choose to do that using the specific options, but I don't think this is necessary in most scenarios. The init container image, driver and executor images can be identical by default.
On Thu, Mar 8, 2018 at 6:52 AM purna pradeep <purna2prad...@gmail.com> wrote: > Im trying to run spark-submit to kubernetes cluster with spark 2.3 docker > container image > > The challenge im facing is application have a mainapplication.jar and > other dependency files & jars which are located in Remote location like AWS > s3 ,but as per spark 2.3 documentation there is something called kubernetes > init-container to download remote dependencies but in this case im not > creating any Podspec to include init-containers in kubernetes, as per > documentation Spark 2.3 spark/kubernetes internally creates Pods > (driver,executor) So not sure how can i use init-container for spark-submit > when there are remote dependencies. > > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-remote-dependencies > > Please suggest > -- Anirudh Ramanathan
