+1 for the idea and feature, but I think the design is definitely lacking
detail on the internal changes needed and how the execution pieces work and the
communication. Are you planning on posting more of those details or were you
just planning on discussing in PR?
Tom
On Wednesday, November 1, 2017, 11:29:21 AM CDT, Debasish Das
<debasish.da...@gmail.com> wrote:
+1
Is there any design doc related to API/internal changes ? Will CP be the
default in structured streaming or it's a mode in conjunction with exisiting
behavior.
Thanks.Deb
On Nov 1, 2017 8:37 AM, "Reynold Xin" <r...@databricks.com> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:
https://issues.apache.org/ jira/browse/SPARK-20928
It is meant to be a very small, surgical change to Structured Streaming to
enable ultra-low latency. This is great timing because we are also designing
and implementing data source API v2. If designed properly, we can have the same
data source API working for both streaming and batch.
Following the SPIP process, I'm putting this SPIP up for a vote.
+1: Let's go ahead and design / implement the SPIP.+0: Don't really care.-1: I
do not think this is a good idea for the following reasons.
