Re: [K8S] Spark initContainer custom bootstrap support for Spark master

2018-08-16 Thread Li Gao
Thanks! We will likely use the second option to customize the bootstrap. On Thu, Aug 16, 2018 at 10:04 AM Yinan Li wrote: > Yes, the init-container has been removed in the master branch. The > init-container was used in 2.3.x only for downloading remote dependencies, > which is now handled by

Re: [K8S] Spark initContainer custom bootstrap support for Spark master

2018-08-16 Thread Yinan Li
Yes, the init-container has been removed in the master branch. The init-container was used in 2.3.x only for downloading remote dependencies, which is now handled by running spark-submit in the driver. If you need to run custom bootstrap scripts using an init-container, the best option would be to

[K8S] Spark initContainer custom bootstrap support for Spark master

2018-08-15 Thread Li Gao
Hi, We've noticed on the latest Master (not Spark 2.3.1 branch), the support for Kubernetes initContainer is no longer there. What would be the path forward if we need to do custom bootstrap actions (i.e. run additional scripts) prior to driver/executor container entering running mode? Thanks,