Re: SPIP: Spark on Kubernetes

We have started discussions about upstreaming and merge strategy at our weekly meetings. The associated github issue is: https://github.com/apache-spark-on-k8s/spark/issues/441 There is general consensus that breaking it up into smaller components will be important for upstream review. Our

Re: SPIP: Spark on Kubernetes

Anirudh (or somebody else familiar with spark-on-k8s), Can you create a short plan on how we would integrate and do code review to merge the project? If the diff is too large it'd be difficult to review and merge in one shot. Once we have a plan we can create subtickets to track the progress.

Re: SPIP: Spark on Kubernetes

The proposal is in the process of being updated to include the details on testing that we have, that Imran pointed out. Please expect an update on the SPARK-18278 . Mridul had a couple of points as well, about exposing an SPI and we've been

Re: SPIP: Spark on Kubernetes

This has passed, hasn't it? On Tue, Aug 15, 2017 at 5:33 PM Anirudh Ramanathan wrote: > Spark on Kubernetes effort has been developed separately in a fork, and > linked back from the Apache Spark project as an experimental backend >

Re: SPIP: Spark on Kubernetes

+1 (non-binding) Regards, Vaquar khan On Mon, Aug 28, 2017 at 5:09 PM, Erik Erlandson wrote: > > In addition to the engineering & software aspects of the native Kubernetes > community project, we have also worked at building out the community, with > the goal of providing

Re: SPIP: Spark on Kubernetes

In addition to the engineering & software aspects of the native Kubernetes community project, we have also worked at building out the community, with the goal of providing the foundation for sustaining engineering on the Kubernetes scheduler back-end. That said, I agree 100% with your point that

Re: SPIP: Spark on Kubernetes

> > In my opinion, the fact that there are nearly no changes to spark-core, > and most of our changes are additive should go to prove that this adds > little complexity to the workflow of the committers. Actually (and somewhat perversely), the otherwise praiseworthy isolation of the Kubernetes

Re: SPIP: Spark on Kubernetes

Subject: Re: SPIP: Spark on Kubernetes I run the example in k8s environment based on vagrant. The driver pod report error: Error: Could not find or load man class org.apache.spark.examples.SparkPi. Use local mode. 获取 Outlook for iOS<https://aka.ms/o0ukef>

Re: SPIP: Spark on Kubernetes

pache.org> Sent: Wednesday, August 23, 2017 5:47:16 AM To: dev@spark.apache.org Subject: Re: SPIP: Spark on Kubernetes +1 (non-binding) I am specifically interested in setting up testing environment for my company's Spark use and also expecting more comprehensive documents on getting developme

Re: SPIP: Spark on Kubernetes

development lifecycle can be done, but hope developer guide can have more information. Thanks Edward Zhang -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22210.html Sent from the Apache Spark Developers List mailing list

Re: SPIP: Spark on Kubernetes

Thank you for your comments Imran. Regarding integration tests, What you inferred from the documentation is correct - Integration tests do not require any prior setup or a Kubernetes cluster to run. Minikube is a single binary that brings up a one-node cluster and exposes the full Kubernetes

Re: SPIP: Spark on Kubernetes

Speaking to integration testing: the integration tests can either attach to an existing cluster, or they can spin up their own minikube cluster to run themselves against. Spark-on-kube can definitely operate without the RSS, as long as spark can find the files it needs using some other

Re: SPIP: Spark on Kubernetes

Overall this looks like a good proposal. I do have some concerns which I'd like to discuss -- please understand I'm taking a "devil's advocate" stance here for discussion, not that I'm giving a -1. My primary concern is about testing and maintenance. My concerns might be addressed if the doc

Re: SPIP: Spark on Kubernetes

this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22197.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe e-mail

Re: SPIP: Spark on Kubernetes

: "dev@spark.apache.org" <dev@spark.apache.org> Subject: Re: SPIP: Spark on Kubernetes There are a fair number of people (myself included) who have interest in making scheduler back-ends fully pluggable. That will represent a significant impact to core spark architecture, with corr

Re: SPIP: Spark on Kubernetes

+1 (non-binding) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22195.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: SPIP: Spark on Kubernetes

There are a fair number of people (myself included) who have interest in making scheduler back-ends fully pluggable. That will represent a significant impact to core spark architecture, with corresponding risk. Adding the kubernetes back-end in a manner similar to the other three back-ends has

Re: SPIP: Spark on Kubernetes

While I definitely support the idea of Apache Spark being able to leverage kubernetes, IMO it is better for long term evolution of spark to expose appropriate SPI such that this support need not necessarily live within Apache Spark code base. It will allow for multiple backends to evolve,

Re: SPIP: Spark on Kubernetes

> >> want to bring Spark workloads along with us. > >> > >> On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 <liyinan...@gmail.com > >> <mailto:liyinan...@gmail.com>> wrote: > >>> > >&g

Fwd: SPIP: Spark on Kubernetes

-- Forwarded message -- From: Timothy Chen <tnac...@gmail.com> Date: Thu, Aug 17, 2017 at 2:48 PM Subject: Re: SPIP: Spark on Kubernetes To: Marcelo Vanzin <van...@cloudera.com> Hi Marcelo, Agree with your points, and I had that same thought around Resource st

Re: SPIP: Spark on Kubernetes

+1 (non-binding) peace o/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: SPIP: Spark on Kubernetes

I have just some very high level knowledge of kubernetes, so I can't really comment on the details of the proposal that relate to it. But I have some comments about other areas of the linked documents: - It's good to know that there's a community behind this effort and mentions of lots of

Re: SPIP: Spark on Kubernetes

to unify on Kubernetes and > >> want to bring Spark workloads along with us. > >> > >> On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 <liyinan...@gmail.com> wrote: > >>> > >>> +1 (non-binding) > >>> > >>> > >&

Re: SPIP: Spark on Kubernetes

2, Andrew Ash <and...@andrewash.com> wrote: >> >> >> >> +1 (non-binding) >> >> >> >> We're moving large amounts of infrastructure from a combination of open >> >> source and homegrown cluster management systems to unify on Kubernet

Re: SPIP: Spark on Kubernetes

>> source and homegrown cluster management systems to unify on Kubernetes > and > >> want to bring Spark workloads along with us. > >> > >> On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 <liyinan...@gmail.com> > wrote: > >>> > >>>

Re: SPIP: Spark on Kubernetes

oads along with us. >>> >>> On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 <liyinan...@gmail.com> >wrote: >>>> >>>> +1 (non-binding) >>>> >>>> >>>> >>>> -- >>>> View this message in context

Re: SPIP: Spark on Kubernetes

; want to bring Spark workloads along with us. >> >> On Tue, Aug 15, 2017 at 2:29 PM, liyinan926 <liyinan...@gmail.com> wrote: >>> >>> +1 (non-binding) >>> >>> >>> >>> -- >>> View this message in context: >>> h

Re: SPIP: Spark on Kubernetes

inan...@gmail.com> wrote: > >> +1 (non-binding) >> >> >> >> -- >> View this message in context: http://apache-spark-developers >> -list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22164.html &g

回复： SPIP: Spark on Kubernetes

+1 在2017年08月16日 04:53，Jiri Kremser 写道： +1 (non-binding) On Tue, Aug 15, 2017 at 10:19 PM, Shubham Chopra wrote: +1 (non-binding) ~Shubham. On Tue, Aug 15, 2017 at 2:11 PM, Erik Erlandson wrote: Kubernetes has

Re: SPIP: Spark on Kubernetes

; +1 (non-binding) > > > > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/SPIP-Spark-on- > Kubernetes-tp22147p22164.html > Sent from the Apache Spark Developers List mai

Re: SPIP: Spark on Kubernetes

+1 (non-binding) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22164.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: SPIP: Spark on Kubernetes

+1 (non-binding) ~Shubham. On Tue, Aug 15, 2017 at 2:11 PM, Erik Erlandson wrote: > > Kubernetes has evolved into an important container orchestration platform; > it has a large and growing user base and an active ecosystem. Users of > Apache Spark who are also deploying

Re: SPIP: Spark on Kubernetes

Kubernetes has evolved into an important container orchestration platform; it has a large and growing user base and an active ecosystem. Users of Apache Spark who are also deploying applications on Kubernetes (or are planning to) will have convergence-related motivations for migrating their Spark

Re: SPIP: Spark on Kubernetes

+1 (non-binding) Glad to see this moving forward :D On Tue, Aug 15, 2017 at 10:10 AM Holden Karau wrote: > +1 (non-binding) > > I (personally) think that Kubernetes as a scheduler backend should > eventually get merged in and there is clearly a community interested in the

Re: SPIP: Spark on Kubernetes

+1 (non-binding) I (personally) think that Kubernetes as a scheduler backend should eventually get merged in and there is clearly a community interested in the work required to maintain it. On Tue, Aug 15, 2017 at 9:51 AM William Benton wrote: > +1 (non-binding) > > On Tue,

Re: SPIP: Spark on Kubernetes

+1 (non-binding) On Tue, Aug 15, 2017 at 10:32 AM, Anirudh Ramanathan < fox...@google.com.invalid> wrote: > Spark on Kubernetes effort has been developed separately in a fork, and > linked back from the Apache Spark project as an experimental backend >

Re: SPIP: Spark on Kubernetes

gt;> >> >> >> -- >> View this message in context: >> http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22150.html >> Sent from the Apache Spark Developers List mailing list archive at >> Nabble.com. >> >

Re: SPIP: Spark on Kubernetes

+1 (non-binding) Thanks, Kimoon On Tue, Aug 15, 2017 at 9:19 AM, Sean Suchter <sean-apachesp...@suchter.com> wrote: > +1 (non-binding) > > > > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/SPIP-Spark-on- > Kubern

Re: SPIP: Spark on Kubernetes

+1 (non-binding) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-tp22147p22150.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: SPIP: Spark on Kubernetes

+1 (non-binding) On Tue, Aug 15, 2017 at 8:32 AM, Anirudh Ramanathan wrote: > Spark on Kubernetes effort has been developed separately in a fork, and > linked back from the Apache Spark project as an experimental backend >