Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1, looking forward to more design details of this feature. Thanks Jerry On Wed, Nov 8, 2017 at 6:40 AM, Shixiong(Ryan) Zhu wrote: > +1 > > On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley > wrote: > >> +1 >> >> On Mon, Nov 6, 2017 at 5:11 PM,

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

The vote has passed with the following +1s: Reynold Xin* Debasish Das Noman Khan Wenchen Fan* Matei Zaharia* Weichen Xu Vaquar Khan Burak Yavuz Xiao Li Tom Graves* Michael Armbrust* Joseph Bradley* Shixiong Zhu* And the following +0s: Sean Owen* Thanks for the feedback! On Wed, Nov 1, 2017

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley wrote: > +1 > > On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust > wrote: > >> +1 >> >> On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li wrote: >> >>> +1 >>> >>> 2017-11-04 11:00

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust wrote: > +1 > > On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li wrote: > >> +1 >> >> 2017-11-04 11:00 GMT-07:00 Burak Yavuz : >> >>> +1 >>> >>> On Fri, Nov 3, 2017 at 10:02 PM, vaquar

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li wrote: > +1 > > 2017-11-04 11:00 GMT-07:00 Burak Yavuz : > >> +1 >> >> On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan >> wrote: >> >>> +1 >>> >>> On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Thanks Tom. I'd imagine more details belong either in a full design doc, or a PR description. Might make sense to do an additional design doc, if there is enough delta from the current sketch doc. On Mon, Nov 6, 2017 at 7:29 AM, Tom Graves wrote: > +1 for the idea and

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 for the idea and feature, but I think the design is definitely lacking detail on the internal changes needed and how the execution pieces work and the communication.  Are you planning on posting more of those details or were you just planning on discussing in PR? Tom On Wednesday,

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan wrote: > +1 > > On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu > wrote: > >> +1. >> >> On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia >> wrote: >> >>> +1 from me too. >>> >>>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu wrote: > +1. > > On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia > wrote: > >> +1 from me too. >> >> Matei >> >> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan wrote: >> > >> > +1.

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1. On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia wrote: > +1 from me too. > > Matei > > > On Nov 3, 2017, at 4:59 PM, Wenchen Fan wrote: > > > > +1. > > > > I think this architecture makes a lot of sense to let executors talk to > source/sink

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 from me too. Matei > On Nov 3, 2017, at 4:59 PM, Wenchen Fan wrote: > > +1. > > I think this architecture makes a lot of sense to let executors talk to > source/sink directly, and bring very low latency. > > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1. I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency. On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen wrote: > +0 simply because I don't feel I know enough to have an opinion. I have no > reason to doubt the

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc. On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin wrote: > Earlier I sent out a discussion thread for CP in Structured Streaming: > >

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 for ultra-low latency Thanks & Regards Noman Get Outlook for Android<https://aka.ms/ghei36> From: Reynold Xin Sent: Wednesday, 1 November, 21:07 Subject: [Vote] SPIP: Continuous Processing Mode for Structured Streaming To: dev@spark.apache.org Earlier I sent out a discuss

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

I just replied. On Wed, Nov 1, 2017 at 5:50 PM, Cody Koeninger wrote: > Was there any answer to my question around the effect of changes to > the sink api regarding access to underlying offsets? > > On Wed, Nov 1, 2017 at 11:32 AM, Reynold Xin wrote: >

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Was there any answer to my question around the effect of changes to the sink api regarding access to underlying offsets? On Wed, Nov 1, 2017 at 11:32 AM, Reynold Xin wrote: > Most of those should be answered by the attached design sketch in the JIRA > ticket. > > On Wed, Nov

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

+1 Is there any design doc related to API/internal changes ? Will CP be the default in structured streaming or it's a mode in conjunction with exisiting behavior. Thanks. Deb On Nov 1, 2017 8:37 AM, "Reynold Xin" wrote: Earlier I sent out a discussion thread for CP in

[Vote] SPIP: Continuous Processing Mode for Structured Streaming

Earlier I sent out a discussion thread for CP in Structured Streaming: https://issues.apache.org/jira/browse/SPARK-20928 It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data