Hi Morrigan, Thanks for joining discussion. I thought we need to hear your goal to donate SQE code, and opinion for how to apply SQE to Storm SQL and working on further improvements.
Not sure when you took a look at the feature set of Storm SQL, but if you haven't recently, you may want to do that. I started working on improving Storm SQL several weeks ago, and many things are addressed in recent weeks. * STORM-1435 <https://issues.apache.org/jira/browse/STORM-1435>: You can easily launch Storm SQL runner without concerning dependencies for Storm SQL core and runtime. It wasn't easy to run before STORM-2016 <http://issues.apache.org/jira/browse/STORM-2016> is introduced. * Refactored Storm SQL code for Trident to fit to Trident operations. Storm SQL parsed SQL and generated topology code but it was not easy to know how topology code is generated, and also hard to determine how Trident optimizations are applied. * STORM-1434 <https://issues.apache.org/jira/browse/STORM-1434>, STORM-2050 <https://issues.apache.org/jira/browse/STORM-2050>: Addressed GROUP BY with UDAF (User Defined Aggregate Function) on Trident mode. Storm SQL already supported UDF on Trident mode. * STORM-2057 <https://issues.apache.org/jira/browse/STORM-2057>: JOIN (inner, left outer, right outer, full outer) feature is now on reviewing. Note that only equi-join is supported. The changes are not included to official release yet, but I expect Storm 1.1.0 will include them which are worth to try out for early adopters. You can also refer STORM-1433 <https://issues.apache.org/jira/browse/STORM-1433> for current phase of Storm SQL. Might need to have another phases (epics) for resolving other issues as well. I only had a look at SQE wiki so don't know the detailed features of SQE, but my feeling is that recent changes fills the gap between SQE and Storm SQL, and even addressing some TODOs of SQE. We might need to cross check feature set of each project to make clear on pros and cons for each project. Btw, while Storm SQL has been implemented its missing features, the difficult part for Storm SQL is SQL optimizations. There seems lots of SQL optimizations (like filter pushdown) but I'm not expert on that and it apparently needs more deep understanding of Calcite. Other parts also need contributors but we strongly need contributors in this area. Thanks, Jungtaek Lim (HeartSaVioR) 2016년 8월 31일 (수) 오전 12:47, Morrigan Jones <[email protected]>님이 작성: > Hi, I'm the original creator and primary developer of SQE. Sorry for > the radio silence on my part, I was out on vacation the past two > weeks. > > I'm glad to see the Storm SQL project chugging along. I started SQE > because I wanted better tools on top of Storm, particularly the > ability to query streams and build topologies using SQL. Our > philosophy is to quickly iterate on our production systems and provide > immediate value. We've been able to do this with SQE, which powers our > streaming systems. Work on SQE and adding functions is driven by our > current use cases. The big near term item on our road map is to add > SQL parsing. Calcite is very promising there and brings lots of > additional features, as I'm sure you know. Additionally, we're going > to improve our function, stream, and state support. > > The difficulty I can see for us with Storm SQL is the amount of work > necessary to get from where we are now with SQE to integrating any > functionality and making sure Storm SQL can provide the functionality > we have now, assuming that is the path we would all go. We're super > excited to see support for Storm grow and mature, and we'd like to be > a part of that. But we also have to maintain our ability to iterate > quickly and provide immediate value. > > > > -- > Morrigan Jones > Principal Engineer > JWPLAYER | Your Way to Play > [email protected] | jwplayer.com >
