On Mon, Oct 8, 2018 at 6:36 AM Rob Vesse <rve...@dotnetrdf.org> wrote: > Since connectivity back to the client is a potential stumbling block for > cluster mode I wander if it would be better to think in reverse i.e. rather > than having the driver pull from the client have the client push to the > driver pod? > > You can do this manually yourself via kubectl cp so it should be possible to > programmatically do this since it looks like this is just a tar piped into a > kubectl exec. This would keep the relevant logic in the Kubernetes specific > client which may/may not be desirable depending on whether we’re looking to > just fix this for K8S or more generally. Of course there is probably a fair > bit of complexity in making this work but does that sound like something > worth exploring?
That sounds like a good solution especially if there's a programmatic API for it, instead of having to fork a sub-process to upload the files. > I hadn’t really considered the HA aspect When you say HA here what do you mean exactly? I don't really expect two drivers for the same app running at the same time, so my first guess is you mean "reattempts" just like YARN supports - re-running the driver if the first one fails? That can be tricky without some shared storage mechanism, because in cluster mode the submission client doesn't need to stay alive after the application starts. Or at least it doesn't with other cluster managers. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
