Re: SPIP: Spark on Kubernetes

While I definitely support the idea of Apache Spark being able to leverage kubernetes, IMO it is better for long term evolution of spark to expose appropriate SPI such that this support need not necessarily live within Apache Spark code base. It will allow for multiple backends to evolve,

Re: SPIP: Spark on Kubernetes

@reynold: Databricks runs their proprietary product on Kubernetes. how about contributing some of that work back to the Open Source Community? — Chris Fregly Founder and Research Engineer @ PipelineAI Founder @ Advanced Spark and TensorFlow Meetup

Fwd: SPIP: Spark on Kubernetes

-- Forwarded message -- From: Timothy Chen Date: Thu, Aug 17, 2017 at 2:48 PM Subject: Re: SPIP: Spark on Kubernetes To: Marcelo Vanzin Hi Marcelo, Agree with your points, and I had that same thought around Resource staging server and

Re: SPIP: Spark on Kubernetes

+1 (non-binding) peace o/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: SPIP: Spark on Kubernetes

I have just some very high level knowledge of kubernetes, so I can't really comment on the details of the proposal that relate to it. But I have some comments about other areas of the linked documents: - It's good to know that there's a community behind this effort and mentions of lots of

Re: SPIP: Spark on Kubernetes

+1 from me as well. Matei > On Aug 17, 2017, at 10:55 AM, Reynold Xin wrote: > > +1 on adding Kubernetes support in Spark (as a separate module similar to how > YARN is done) > > I talk with a lot of developers and teams that operate cloud services, and > k8s in the

Re: SPIP: Spark on Kubernetes

+1 on adding Kubernetes support in Spark (as a separate module similar to how YARN is done) I talk with a lot of developers and teams that operate cloud services, and k8s in the last year has definitely become one of the key projects, if not the one with the strongest momentum in this space. I'm

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Points 2, 3 and 4 of the Project Plan in that document (i.e. "port existing data sources using internal APIs to use the proposed public Data Source V2 API") have my full support. Really, I'd like to see that dog-fooding effort completed and lesson learned from it fully digested before we remove

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Yea I don't think it's a good idea to upload a doc and then call for a vote immediately. People need time to digest ... On Thu, Aug 17, 2017 at 6:22 AM, Wenchen Fan wrote: > Sorry let's remove the VOTE tag as I just wanna bring this up for > discussion. > > I'll restart

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Sorry let's remove the VOTE tag as I just wanna bring this up for discussion. I'll restart the voting process after we have enough discussion on the JIRA ticket or here in this email thread. On Thu, Aug 17, 2017 at 9:12 PM, Russell Spitzer wrote: > -1, I don't think

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

-1, I don't think there has really been any discussion of this api change yet or at least it hasn't occurred on the jira ticket On Thu, Aug 17, 2017 at 8:05 AM Wenchen Fan wrote: > adding my own +1 (binding) > > On Thu, Aug 17, 2017 at 9:02 PM, Wenchen Fan

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

+1 (non-binding) Wenchen Fan 于2017年8月17日 周四下午9:05写道： > adding my own +1 (binding) > > On Thu, Aug 17, 2017 at 9:02 PM, Wenchen Fan wrote: > >> Hi all, >> >> Following the SPIP process, I'm putting this SPIP up for a vote. >> >> The current data source

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

adding my own +1 (binding) On Thu, Aug 17, 2017 at 9:02 PM, Wenchen Fan wrote: > Hi all, > > Following the SPIP process, I'm putting this SPIP up for a vote. > > The current data source API doesn't work well because of some limitations > like: no partitioning/bucketing

[VOTE] [SPIP] SPARK-15689: Data Source API V2

Hi all, Following the SPIP process, I'm putting this SPIP up for a vote. The current data source API doesn't work well because of some limitations like: no partitioning/bucketing support, no columnar read, hard to support more operator push down, etc. I'm proposing a Data Source API V2 to