Re: [K8S] Spark initContainer custom bootstrap support for Spark master
Thanks! We will likely use the second option to customize the bootstrap. On Thu, Aug 16, 2018 at 10:04 AM Yinan Li wrote: > Yes, the init-container has been removed in the master branch. The > init-container was used in 2.3.x only for downloading remote dependencies, > which is now handled by running spark-submit in the driver. If you need to > run custom bootstrap scripts using an init-container, the best option would > be to use a mutating admission webhook to inject your init-container into > the Spark pods. Another option is to create a custom image that runs the > scripts prior to entering the entrypoint. > > Yinan > > On Wed, Aug 15, 2018 at 9:12 AM Li Gao wrote: > >> Hi, >> >> We've noticed on the latest Master (not Spark 2.3.1 branch), the support >> for Kubernetes initContainer is no longer there. What would be the path >> forward if we need to do custom bootstrap actions (i.e. run additional >> scripts) prior to driver/executor container entering running mode? >> >> Thanks, >> Li >> >>
Re: [K8S] Spark initContainer custom bootstrap support for Spark master
Yes, the init-container has been removed in the master branch. The init-container was used in 2.3.x only for downloading remote dependencies, which is now handled by running spark-submit in the driver. If you need to run custom bootstrap scripts using an init-container, the best option would be to use a mutating admission webhook to inject your init-container into the Spark pods. Another option is to create a custom image that runs the scripts prior to entering the entrypoint. Yinan On Wed, Aug 15, 2018 at 9:12 AM Li Gao wrote: > Hi, > > We've noticed on the latest Master (not Spark 2.3.1 branch), the support > for Kubernetes initContainer is no longer there. What would be the path > forward if we need to do custom bootstrap actions (i.e. run additional > scripts) prior to driver/executor container entering running mode? > > Thanks, > Li > >
[K8S] Spark initContainer custom bootstrap support for Spark master
Hi, We've noticed on the latest Master (not Spark 2.3.1 branch), the support for Kubernetes initContainer is no longer there. What would be the path forward if we need to do custom bootstrap actions (i.e. run additional scripts) prior to driver/executor container entering running mode? Thanks, Li