Wanted to know active branch for the beam SQL feature, is it beam - 301 where the development for this feature would happen?
Regards, Tarush On Tue, 11 Apr 2017 at 10:49 AM, 陈竞 <cj.mag...@gmail.com> wrote: > i just want to know what the SQL State API equivalent is for SQL, since > beam has already support stateful processing using state DoFn > > 2017-04-11 2:12 GMT+08:00 Tyler Akidau <taki...@google.com.invalid>: > > > 陈竞, what are you specifically curious about regarding state? Are you > > wanting to know what the SQL State API equivalent is for SQL? Or are you > > asking an operational question about where the state for a given SQL > > pipeline will live? > > > > -Tyler > > > > > > On Sun, Apr 9, 2017 at 12:39 PM Mingmin Xu <mingm...@gmail.com> wrote: > > > > > Thanks @JB, will come out the initial PR soon. > > > > > > On Sun, Apr 9, 2017 at 12:28 PM, Jean-Baptiste Onofré <j...@nanthrax.net > > > > > wrote: > > > > > > > As discussed, I created the DSL_SQL branch with the skeleton. Mingmin > > is > > > > rebasing on this branch to submit the PR. > > > > > > > > Regards > > > > JB > > > > > > > > > > > > On 04/09/2017 08:02 PM, Mingmin Xu wrote: > > > > > > > >> State is not touched yet, welcome to add it. > > > >> > > > >> On Sun, Apr 9, 2017 at 2:40 AM, 陈竞 <cj.mag...@gmail.com> wrote: > > > >> > > > >> how will this sql support state both in streaming and batch mode > > > >>> > > > >>> 2017-04-07 4:54 GMT+08:00 Mingmin Xu <mingm...@gmail.com>: > > > >>> > > > >>> @Tyler, there's no big change in the previous design doc, I added > > some > > > >>>> details in chapter 'Part 2. DML( [INSERT] SELECT )' , describing > > steps > > > >>>> to > > > >>>> process a query, feel free to leave a comment. > > > >>>> > > > >>>> Come through your doc of 'EMIT', it's awesome from my perspective. > > > I've > > > >>>> some tests on GroupBy with default triggers/allowed_lateness now. > > EMIT > > > >>>> syntax can be added to fill the gap. > > > >>>> > > > >>>> On Thu, Apr 6, 2017 at 1:04 PM, Tyler Akidau <taki...@apache.org> > > > >>>> wrote: > > > >>>> > > > >>>> I'm very excited by this development as well, thanks for > continuing > > to > > > >>>>> > > > >>>> push > > > >>>> > > > >>>>> this forward, Mingmin. :-) > > > >>>>> > > > >>>>> I noticed you'd made some changes to your design doc > > > >>>>> < > https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_ > > > >>>>> 0a1Bz5BsCROMzCU/edit>. > > > >>>>> Is it ready for another review? How reflective is it currently of > > the > > > >>>>> > > > >>>> work > > > >>>> > > > >>>>> that going into the feature branch? > > > >>>>> > > > >>>>> In parallel, I'd also like to continue helping push forward the > > > >>>>> > > > >>>> definition > > > >>>> > > > >>>>> of unified model semantics for SQL so we can get Calcite to a > point > > > >>>>> > > > >>>> where > > > >>> > > > >>>> it supports the full Beam model. I added a comment > > > >>>>> <https://issues.apache.org/jira/browse/BEAM-301? > > > >>>>> > > > >>>> focusedCommentId=15959621& > > > >>>> > > > >>>>> page=com.atlassian.jira.plugin.system.issuetabpanels: > > > >>>>> comment-tabpanel#comment-15959621> > > > >>>>> on the JIRA suggesting I create a doc with a specification > proposal > > > for > > > >>>>> EMIT (and any other necessary semantic changes) that we can then > > > >>>>> > > > >>>> iterate > > > >>> > > > >>>> on > > > >>>> > > > >>>>> in public with the Calcite folks. I already have most of the > > content > > > >>>>> written (and there's a significant amount of background needed to > > > >>>>> > > > >>>> justify > > > >>> > > > >>>> some aspects of the proposal), so it'll mostly be a matter of > > pulling > > > >>>>> > > > >>>> it > > > >>> > > > >>>> all together into something coherent. Does that sound reasonable > to > > > >>>>> everyone? > > > >>>>> > > > >>>>> -Tyler > > > >>>>> > > > >>>>> > > > >>>>> On Thu, Apr 6, 2017 at 10:26 AM Kenneth Knowles > > > <k...@google.com.invalid > > > >>>>> > > > >>>> > > > >>>> wrote: > > > >>>>> > > > >>>>> Very cool! I'm really excited about this integration. > > > >>>>>> > > > >>>>>> On Thu, Apr 6, 2017 at 9:39 AM, Jean-Baptiste Onofré < > > > >>>>>> > > > >>>>> j...@nanthrax.net> > > > >>> > > > >>>> wrote: > > > >>>>>> > > > >>>>>> Hi, > > > >>>>>>> > > > >>>>>>> Mingmin and I prepared a new branch to have the SQL DSL in > > dsls/sql > > > >>>>>>> location. > > > >>>>>>> > > > >>>>>>> Any help is welcome ! > > > >>>>>>> > > > >>>>>>> Thanks, > > > >>>>>>> Regards > > > >>>>>>> JB > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On 04/06/2017 06:36 PM, Mingmin Xu wrote: > > > >>>>>>> > > > >>>>>>> @Tarush, you're very welcome to join the effort. > > > >>>>>>>> > > > >>>>>>>> On Thu, Apr 6, 2017 at 7:22 AM, tarush grover < > > > >>>>>>>> > > > >>>>>>> tarushappt...@gmail.com> > > > >>>>> > > > >>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>> Hi, > > > >>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> Can I be also part of this feature development. > > > >>>>>>>>> > > > >>>>>>>>> Regards, > > > >>>>>>>>> Tarush Grover > > > >>>>>>>>> > > > >>>>>>>>> On Thu, Apr 6, 2017 at 3:17 AM, Ted Yu <yuzhih...@gmail.com> > > > >>>>>>>>> > > > >>>>>>>> wrote: > > > >>>> > > > >>>>> > > > >>>>>>>>> I compiled BEAM-301 branch with calcite 1.12 - passed. > > > >>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> Julian tries to not break existing things, but he will if > > > >>>>>>>>>> > > > >>>>>>>>> there's > > > >>> > > > >>>> a > > > >>>> > > > >>>>> > > > >>>>>>>>>> reason > > > >>>>>>>>> > > > >>>>>>>>> to do so :-) > > > >>>>>>>>>> > > > >>>>>>>>>> On Wed, Apr 5, 2017 at 2:36 PM, Mingmin Xu < > > mingm...@gmail.com> > > > >>>>>>>>>> > > > >>>>>>>>> wrote: > > > >>>>>> > > > >>>>>>> > > > >>>>>>>>>> @Ted, thanks for the note. I intend to stick with one > version, > > > >>>>>>>>>> > > > >>>>>>>>> Beam > > > >>>> > > > >>>>> > > > >>>>>>>>>>> 0.6.0 > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> and Calcite 1.11 so far, unless impacted by API change. > Before > > > >>>>>>>>>> > > > >>>>>>>>> it's > > > >>>> > > > >>>>> > > > >>>>>>>>>>> merged > > > >>>>>>>>>> > > > >>>>>>>>>> back to master, will upgrade to the latest version. > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Wed, Apr 5, 2017 at 2:14 PM, Ted Yu < > yuzhih...@gmail.com> > > > >>>>>>>>>>> > > > >>>>>>>>>> wrote: > > > >>>>> > > > >>>>>> > > > >>>>>>>>>>> Working in feature branch is good - you may want to > > > >>>>>>>>>>> > > > >>>>>>>>>> periodically > > > >>> > > > >>>> sync > > > >>>>> > > > >>>>>> > > > >>>>>>>>>>>> up > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> with master. > > > >>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I noticed that you are using 1.11.0 of calcite. > > > >>>>>>>>>>>> 1.12 is out, FYI > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Wed, Apr 5, 2017 at 2:05 PM, Mingmin Xu < > > > >>>>>>>>>>>> > > > >>>>>>>>>>> mingm...@gmail.com> > > > >>> > > > >>>> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>>> Hi all, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> I'm working on https://issues.apache.org/ > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> jira/browse/BEAM-301(Add > > > >>>>> > > > >>>>>> > > > >>>>>>>>>>>>> a > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> Beam > > > >>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> SQL DSL). The skeleton is already in > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> https://github.com/XuMingmin/beam/tree/BEAM-301, using > > Java > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> SDK > > > >>>> > > > >>>>> in > > > >>>>> > > > >>>>>> > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> back-end. The goal is to provide a SQL interface over Beam, > > > >>>>>>>>>>> > > > >>>>>>>>>> based > > > >>> > > > >>>> > > > >>>>>>>>>>>>> on > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> Calcite, including: > > > >>>>>>>>>> > > > >>>>>>>>>>> 1). a translator to create Beam pipeline from SQL, > > > >>>>>>>>>>>>> (SELECT/INSERT/FILTER/GROUP-BY/JOIN/...); > > > >>>>>>>>>>>>> 2). an interactive client to submit queries; (All-SQL > > mode) > > > >>>>>>>>>>>>> 3). a SQL API which reduce the work to create a Pipeline; > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> (Semi-SQL > > > >>>>> > > > >>>>>> > > > >>>>>>>>>>>>> mode) > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>>> As we see many folks are interested in this feature, would > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> like > > > >>> > > > >>>> to > > > >>>>> > > > >>>>>> > > > >>>>>>>>>>>>> create a > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> feature branch to have more involvement. > > > >>>>>>>>>>>>> Looking for comments and feedback. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks! > > > >>>>>>>>>>>>> ---- > > > >>>>>>>>>>>>> Mingmin > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> -- > > > >>>>>>>>>>> ---- > > > >>>>>>>>>>> Mingmin > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> -- > > > >>>>>>> Jean-Baptiste Onofré > > > >>>>>>> jbono...@apache.org > > > >>>>>>> http://blog.nanthrax.net > > > >>>>>>> Talend - http://www.talend.com > > > >>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> ---- > > > >>>> Mingmin > > > >>>> > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> 陈竞，中科院计算技术研究所，高性能计算机中心 > > > >>> Jing Chen HPCC.ICT.AC China > > > >>> > > > >>> > > > >> > > > >> > > > >> > > > > -- > > > > Jean-Baptiste Onofré > > > > jbono...@apache.org > > > > http://blog.nanthrax.net > > > > Talend - http://www.talend.com > > > > > > > > > > > > > > > > -- > > > ---- > > > Mingmin > > > > > > > > > -- > 陈竞，中科院计算技术研究所，高性能计算机中心 > Jing Chen HPCC.ICT.AC China >
