Thanks Tom. I'd imagine more details belong either in a full design doc, or a PR description. Might make sense to do an additional design doc, if there is enough delta from the current sketch doc.
On Mon, Nov 6, 2017 at 7:29 AM, Tom Graves <tgraves...@yahoo.com> wrote: > +1 for the idea and feature, but I think the design is definitely lacking > detail on the internal changes needed and how the execution pieces work and > the communication. Are you planning on posting more of those details or > were you just planning on discussing in PR? > > Tom > > On Wednesday, November 1, 2017, 11:29:21 AM CDT, Debasish Das < > debasish.da...@gmail.com> wrote: > > > +1 > > Is there any design doc related to API/internal changes ? Will CP be the > default in structured streaming or it's a mode in conjunction with > exisiting behavior. > > Thanks. > Deb > > On Nov 1, 2017 8:37 AM, "Reynold Xin" <r...@databricks.com> wrote: > > Earlier I sent out a discussion thread for CP in Structured Streaming: > > https://issues.apache.org/ jira/browse/SPARK-20928 > <https://issues.apache.org/jira/browse/SPARK-20928> > > It is meant to be a very small, surgical change to Structured Streaming to > enable ultra-low latency. This is great timing because we are also > designing and implementing data source API v2. If designed properly, we can > have the same data source API working for both streaming and batch. > > > Following the SPIP process, I'm putting this SPIP up for a vote. > > +1: Let's go ahead and design / implement the SPIP. > +0: Don't really care. > -1: I do not think this is a good idea for the following reasons. > > > >
