Re: [DISCUSS][K8S] Local dependencies with Kubernetes

From: Rob Vesse Date: Monday, October 8, 2018 at 10:09 AM To: dev Subject: Re: [DISCUSS][K8S] Local dependencies with Kubernetes Well yes. However the submission client is already able to monitor the driver pod status so can see when it is up and running. And couldn’t we potentially modify

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

? I guess at this stage I am just throwing ideas out there and trying to figure out what’s practical/reasonable Rob From: Yinan Li Date: Monday, 8 October 2018 at 17:36 To: Rob Vesse Cc: dev Subject: Re: [DISCUSS][K8S] Local dependencies with Kubernetes However, the pod must be up

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

first step would be to get the > basics working and then look at the HA aspect. Although if the above > theoretical approach is practical that could simply be part of restarting > the driver. > > > > Rob > > > > > > *From: *Felix Cheung > *Date: *Sunday,

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

On Mon, Oct 8, 2018 at 6:36 AM Rob Vesse wrote: > Since connectivity back to the client is a potential stumbling block for > cluster mode I wander if it would be better to think in reverse i.e. rather > than having the driver pull from the client have the client push to the > driver pod? > >

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

. Rob From: Felix Cheung Date: Sunday, 7 October 2018 at 23:00 To: Yinan Li , Stavros Kontopoulos Cc: Rob Vesse , dev Subject: Re: [DISCUSS][K8S] Local dependencies with Kubernetes Jars and libraries only accessible locally at the driver is fairly limited? Don’t you want the same

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

Jars and libraries only accessible locally at the driver is fairly limited? Don’t you want the same on all executor? From: Yinan Li Sent: Friday, October 5, 2018 11:25 AM To: Stavros Kontopoulos Cc: rve...@dotnetrdf.org; dev Subject: Re: [DISCUSS][K8S] Local

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

> Just to be clear: in client mode things work right? (Although I'm not really familiar with how client mode works in k8s - never tried it.) If the driver runs on the submission client machine, yes, it should just work. If the driver runs in a pod, however, it faces the same problem as in cluster

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

@Marcelo is correct. Mesos does not have something similar. Only Yarn does due to the distributed cache thing. I have described most of the above in the the jira also there are some other options. Best, Stavros On Fri, Oct 5, 2018 at 8:28 PM, Marcelo Vanzin wrote: > On Fri, Oct 5, 2018 at 7:54

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

Agreed with Marcelo that this is not a unique problem to Spark on k8s. For a lot of organizations, hosting dependencies on HDFS seems the choice. One option that the Spark Operator does is to automatically upload

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

On Fri, Oct 5, 2018 at 7:54 AM Rob Vesse wrote: > Ideally this would all just be handled automatically for users in the way > that all other resource managers do I think you're giving other resource managers too much credit. In cluster mode, only YARN really distributes local dependencies,

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

Hi Rob, Interesting topic and affects UX a lot. I provided my thoughts in the related jira. Best, Stavros On Fri, Oct 5, 2018 at 5:53 PM, Rob Vesse wrote: > Folks > > > > One of the big limitations of the current Spark on K8S implementation is > that it isn’t possible to use local

[DISCUSS][K8S] Local dependencies with Kubernetes

Folks One of the big limitations of the current Spark on K8S implementation is that it isn’t possible to use local dependencies (SPARK-23153 [1]) i.e. code, JARs, data etc that only lives on the submission client.  This basically leaves end users with several options on how to actually run