Hi, Timo: https://docs.google.com/document/d/18XbrFuwzzRzHR84a48j17BIGpTndSS1q-shSgCBEd8I/edit?usp=sharing is a small design doc for the implementation. I agree with you to put SQL client integration in another PR and focus on the sink implementation in this PR.
On Mon, Mar 5, 2018 at 9:33 PM Timo Walther <twal...@apache.org> wrote: > Hi Renjie, > > it would be great if we can reuse my QueryableStateTableSink > implementation. As far as I know, the queryable state interfaces changed > a bit in the last months and a new queryable state client as been added. > So my branch needs to be rebased to the current APIs. > > We don't need a big design document but at least some paragraphs how the > APIs should look like. How do you want to extend the SQL client for > that? We can also introduce the QueryableStateTableSink in one pull > request and do the API in a follow-up PR. > > What do you think? > > Regards, > Timo > > > Am 3/5/18 um 3:52 AM schrieb Renjie Liu: > > Hi, Timo: > > I've read your QueryableStateTableSink implementation and that basically > > implementes what I want to do. I also want to extend SQL client so that > > user can do point query against the table sink. Do we still need a design > > doc for that? It seems that I just need to finish the left part and do > some > > test against it. > > > > Hi, Stefano: > > Your requirement needs some changes to the flink table implementation > but I > > don't know why you need that? For debugging? The operator state is > internal > > and subject to optisimation logic, so I think it maybe meanless to expose > > that. > > > > On Fri, Mar 2, 2018 at 9:37 PM Stefano Bortoli < > stefano.bort...@huawei.com> > > wrote: > > > >> Hi Timo, Renjie, > >> > >> What I was thinking did not include the QueryableStateTableSink, but > >> rather tap in directly into the state of a streaming operator. Perhaps > it > >> is the same thing, but just it sounds not intuitive to consider it a > sink. > >> > >> So, we would need a way to configure the environment for the query to > >> share the "state name" before the query is executed, and then use this > to > >> create the hook for the queriable state in the operator. Perhaps extend > the > >> current codegen and operator implementations to get as a parameter the > >> StateDescriptor to be inquired. > >> > >> Looking forward for the design document, will be happy to give you > >> feedback. > >> > >> Best, > >> Stefano > >> > >> -----Original Message----- > >> From: Renjie Liu [mailto:liurenjie2...@gmail.com] > >> Sent: Friday, March 02, 2018 11:42 AM > >> To: dev@flink.apache.org > >> Subject: Re: StreamSQL queriable state > >> > >> Great, thank you. > >> I'll start by writing a design doc. > >> > >> On Fri, Mar 2, 2018 at 6:40 PM Timo Walther <twal...@apache.org> wrote: > >> > >>> I gave you contributor permissions in Jira. You should be able to > >>> assign it to yourself now. > >>> > >>> Am 3/2/18 um 11:33 AM schrieb Renjie Liu: > >>>> Hi, Timo: > >>>> It seems that I can't assign it to myself. Could you please help to > >>> assign > >>>> that to me? > >>>> My jira username is liurenjie1024 and my email is > >>> liurenjie2...@gmail.com > >>>> On Fri, Mar 2, 2018 at 6:24 PM Timo Walther <twal...@apache.org> > >> wrote: > >>>>> Hi Renjie, > >>>>> > >>>>> that would be great. There is already a Jira issue for it: > >>>>> https://issues.apache.org/jira/browse/FLINK-6968 > >>>>> > >>>>> Feel free to assign it to yourself. You can reuse parts of my code > >>>>> if you want. But maybe it would make sense to have a little design > >>>>> document first about what we want to support. > >>>>> > >>>>> Regards, > >>>>> Timo > >>>>> > >>>>> > >>>>> Am 3/2/18 um 11:10 AM schrieb Renjie Liu: > >>>>>> Hi, Timo, I've been planning on the same thing and would like to > >>>>> contribute > >>>>>> that. > >>>>>> > >>>>>> On Fri, Mar 2, 2018 at 6:05 PM Timo Walther <twal...@apache.org> > >>> wrote: > >>>>>>> Hi Stefano, > >>>>>>> > >>>>>>> yes there are plan in this direction. Actually, I already worked > >>>>>>> on > >>> such > >>>>>>> a QueryableStateTableSink [1] in the past but never finished it > >>> because > >>>>>>> of priority shifts. Would be great if somebody wants to > >>>>>>> contribute > >>> this > >>>>>>> functionality :) > >>>>>>> > >>>>>>> Regards, > >>>>>>> Timo > >>>>>>> > >>>>>>> [1] https://github.com/twalthr/flink/tree/QueryableTableSink > >>>>>>> > >>>>>>> Am 3/2/18 um 10:58 AM schrieb Stefano Bortoli: > >>>>>>>> Hi guys, > >>>>>>>> > >>>>>>>> I am checking out the queriable state API, and it seems that > >>>>>>>> most of > >>>>> the > >>>>>>> tooling is already available. However, the queriable state is > >>> available > >>>>>>> just for the streaming API, not at the StreamSQL API level. In > >>>>> principle, > >>>>>>> as the flink-table is aware of the query semantic and data output > >>> type, > >>>>> it > >>>>>>> should be possible to configure the query compilation to nest > >>> queriable > >>>>>>> state in the process/window functions. Is there any plan in this > >>>>> direction? > >>>>>>>> Best, > >>>>>>>> Stefano > >>>>>>>> > >>>>>>> -- > >>>>>> Liu, Renjie > >>>>>> Software Engineer, MVAD > >>>>>> > >>>>> -- > >>>> Liu, Renjie > >>>> Software Engineer, MVAD > >>>> > >>> -- > >> Liu, Renjie > >> Software Engineer, MVAD > >> > > -- Liu, Renjie Software Engineer, MVAD
