Re: Kubernetes backend and docker images

Think we can allow for different images and default to them being the same. Apologize if I missed that as being the original intention though. -Matt Cheah On 1/8/18, 1:45 PM, "Marcelo Vanzin" wrote: On Mon, Jan 8, 2018 at 1:39 PM, Matt Cheah

Re: Kubernetes backend and docker images

On Mon, Jan 8, 2018 at 1:39 PM, Matt Cheah wrote: > We would still want images to be able to be uniquely specified for the > driver vs. the executors. For example, not all of the libraries required on > the driver may be required on the executors, so the user would want to >

Re: Kubernetes backend and docker images

// Fixing Anirudh's email address From: Matt Cheah Sent: Monday, January 8, 2018 1:39:12 PM To: Anirudh Ramanathan; Felix Cheung Cc: 蒋星博; Marcelo Vanzin; dev; Timothy Chen Subject: Re: Kubernetes backend and docker images We would still want images to be able

Re: Kubernetes backend and docker images

alantir.com>, Timothy Chen <tnac...@gmail.com> Subject: Re: Kubernetes backend and docker images +matt +tim For reference - here's our previous thread on this dockerfile unification problem - https://github.com/apache-spark-on-k8s/spark/pull/60[github.com] I think this approach should

Re: Kubernetes backend and docker images

t;> we should have one docker image instead of 3 (also pending in the fork are >> python and R variant, we should consider having one that we official >> release instead of 9, for example) >> >> >> -- >> *From:* 蒋星博 <jiangxb

Re: Kubernetes backend and docker images

ing one that we official > release instead of 9, for example) > > > -- > *From:* 蒋星博 <jiangxb1...@gmail.com> > *Sent:* Friday, January 5, 2018 10:57:53 PM > *To:* Marcelo Vanzin > *Cc:* dev > *Subject:* Re: Kubernetes backend and do

Re: Kubernetes backend and docker images

) From: 蒋星博 <jiangxb1...@gmail.com> Sent: Friday, January 5, 2018 10:57:53 PM To: Marcelo Vanzin Cc: dev Subject: Re: Kubernetes backend and docker images Agree it should be nice to have this simplification, and users can still create their custom images by copy/mod

Re: Kubernetes backend and docker images

Agree it should be nice to have this simplification, and users can still create their custom images by copy/modifying the default one. Thanks for bring this out Marcelo! 2018-01-05 17:06 GMT-08:00 Marcelo Vanzin : > Hey all, especially those working on the k8s stuff. > >

Re: Kubernetes backend and docker images

Awesome, less is better Mridul Muralidharan 于2018年1月6日周六 上午11:54写道： > > We should definitely clean this up and make it the default, nicely done > Marcelo ! > > Thanks, > Mridul > > On Fri, Jan 5, 2018 at 5:06 PM Marcelo Vanzin wrote: > >> Hey all,

Re: Kubernetes backend and docker images

We should definitely clean this up and make it the default, nicely done Marcelo ! Thanks, Mridul On Fri, Jan 5, 2018 at 5:06 PM Marcelo Vanzin wrote: > Hey all, especially those working on the k8s stuff. > > Currently we have 3 docker images that need to be built and

Re: Kubernetes backend and docker images

This is neat. With some code cleanup and as long as users can still use custom driver/executor/init-container images if they want to, I think this is great to have. I don't think there's a particular reason why having a single image wouldn't work. Thanks for doing this! On Fri, Jan 5, 2018 at

Kubernetes backend and docker images

Hey all, especially those working on the k8s stuff. Currently we have 3 docker images that need to be built and provided by the user when starting a Spark app: driver, executor, and init container. When the initial review went by, I asked why do we need 3, and I was told that's because they have