[ https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959621#comment-15959621 ]
Tyler Akidau commented on BEAM-301: ----------------------------------- I agree with what I think Mingmin is saying here in that it's important to clearly distinguish between streams and tables. Though it's easy to interchange between them, they should not be treated as 1:1 equivalents. In a situation where you clearly have a source of one type of the other, that should dictate which type of primitive you start out with. That said, I agree with Julian that STREAM, if kept around, is probably just an alternative way of specifying an EMIT clause that emits upon every new INSERT/UPDATE/DELETE. I think EMIT EARLY is probably too general, though. EMIT CHANGELOG, EMIT INSERTS, or even EMIT STREAM (though only because of the history of the STREAM keyword) are all little more specific. But we don't need to bikeshed terms here. I also agree that we should come to some agreement on a full specification for EMIT before forging ahead with any implementations (and to be clear, that's independent from the core SQL DSL stuff you're already putting in the feature branch, Mingmin; that work can proceed unimpeded while we sort out unified model semantics). In that vein, I've dedicated two chapters of my upcoming streaming systems book to the topic (one for the necessary background, and one on SQL specifically) as I've tried to sort the question out for myself. The book won't be out until later this summer at the earliest, though, so maybe in parallel I should try to condense that all into a specification doc we could iterate on in public? I'm not saying what I have there is necessarily the right answer, but it incorporates everything we've more or less agreed upon so far and then extends it a little further, so I think it's probably a good place to start from. I has triggering semantics via EMIT, the semantics necessary for temporal joins, and also addresses things like CUBE in a clean fashion. If we can come to agreement on a way forward in both the Beam and Calcite camps, then we're probably in a good position to forge ahead with implementation details. Sound reasonable? > Add a Beam SQL DSL > ------------------ > > Key: BEAM-301 > URL: https://issues.apache.org/jira/browse/BEAM-301 > Project: Beam > Issue Type: New Feature > Components: sdk-ideas > Reporter: Jean-Baptiste Onofré > Assignee: Xu Mingmin > > The SQL DSL helps developers to build a Beam pipeline from SQL statement in > String directly. > In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one > example SQL as below: > {code} > INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`) > (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, > `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP` > FROM `USEREVENT` AS `USEREVENT` > WHERE `USEREVENT`.`SITEID` > 10) > {code} > A design doc is available at > https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)