Sorry for taking so long I've added a section about Flink Disaggregated State Management Evolution in the attached doc.
I found some of the contents might be overlapped with the "large-scale streaming jobs". So that part might need some changes as well. Please let me know what you think. Best Yuan On Mon, Jul 24, 2023 at 12:07 PM Yuan Mei <yuanmei.w...@gmail.com> wrote: > Sorry have missed this email and respond a bit late. > > I will put a draft for the long-term vision for the state as well as > large-scale state support into the roadmap. > > Best > Yuan > > On Mon, Jul 17, 2023 at 10:34 AM Jark Wu <imj...@gmail.com> wrote: > >> Hi Jiabao, >> >> Thank you for your suggestions. I have added them to the "Going Beyond a >> SQL Stream/Batch Processing Engine" and "Large-Scale State Jobs" sections. >> >> Best, >> Jark >> >> On Thu, 13 Jul 2023 at 16:06, Jiabao Sun <jiabao....@xtransfer.cn >> .invalid> >> wrote: >> >> > Thanks Jark and Martijn for driving this. >> > >> > There are two suggestions about the Table API: >> > >> > - Add the JSON type to adapt to the no sql database type. >> > - Remove changelog normalize operator for upsert stream. >> > >> > >> > Best, >> > Jiabao >> > >> > >> > > 2023年7月13日 下午3:49，Jark Wu <imj...@gmail.com> 写道： >> > > >> > > Hi all, >> > > >> > > Sorry for taking so long back here. >> > > >> > > Martijn and I have drafted the first version of the updated roadmap, >> > > including the updated feature radar reflecting the current state of >> > > different components. >> > > >> > >> https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit >> > > >> > > Feel free to leave comments in the thread or the document. >> > > We may miss mentioning something important, so your help in enriching >> > > the content is greatly appreciated. >> > > >> > > Best, >> > > Jark & Martijn >> > > >> > > >> > > On Fri, 2 Jun 2023 at 00:50, Jing Ge <j...@ververica.com.invalid> >> wrote: >> > > >> > >> Hi Jark, >> > >> >> > >> Fair enough. Let's do it like you suggested. Thanks! >> > >> >> > >> Best regards, >> > >> Jing >> > >> >> > >> On Thu, Jun 1, 2023 at 6:00 PM Jark Wu <imj...@gmail.com> wrote: >> > >> >> > >>> Hi Jing, >> > >>> >> > >>> This thread is for discussing the roadmap for versions 1.18, 2.0, >> and >> > >> even >> > >>> more. >> > >>> One of the outcomes of this discussion will be an updated version of >> > the >> > >>> current roadmap. >> > >>> Let's work together on refining the roadmap in this thread. >> > >>> >> > >>> Best, >> > >>> Jark >> > >>> >> > >>> On Thu, 1 Jun 2023 at 23:25, Jing Ge <j...@ververica.com.invalid> >> > wrote: >> > >>> >> > >>>> Hi Jark, >> > >>>> >> > >>>> Thanks for driving it! For point 2, since we are developing 1.18 >> now, >> > >>>> does it make sense to update the roadmap this time while we are >> > >> releasing >> > >>>> 1.18? This discussion thread will be focusing on the Flink 2.0 >> > roadmap, >> > >>> as >> > >>>> you mentioned previously. WDYT? >> > >>>> >> > >>>> Best regards, >> > >>>> Jing >> > >>>> >> > >>>> On Thu, Jun 1, 2023 at 3:31 PM Jark Wu <imj...@gmail.com> wrote: >> > >>>> >> > >>>>> Hi all, >> > >>>>> >> > >>>>> Martijn and I would like to initiate a discussion on the Flink >> > >> roadmap, >> > >>>>> which should cover the project's long-term roadmap and the regular >> > >>> update >> > >>>>> mechanism. >> > >>>>> >> > >>>>> Xintong has already started a discussion about Flink 2.0 planning. >> > >> One >> > >>> of >> > >>>>> the points raised in that discussion is that we should have a >> > >>> high-level >> > >>>>> discussion of the roadmap to present where the project is heading >> > >>> (which >> > >>>>> doesn't necessarily need to block the Flink 2.0 planning). >> Moreover, >> > >>> the >> > >>>>> roadmap on the Flink website [1] hasn't been updated for half a >> year, >> > >>> and >> > >>>>> the last update was for the feature radar for the 1.15 release. It >> > >> has >> > >>>> been >> > >>>>> 2 years since the community discussed Flink's overall roadmap. >> > >>>>> >> > >>>>> I would like to raise two topics for discussion: >> > >>>>> >> > >>>>> 1. The new roadmap. This should be an updated version of the >> current >> > >>>>> roadmap[1]. >> > >>>>> 2. A mechanism to regularly discuss and update the roadmap. >> > >>>>> >> > >>>>> To make the first topic discussion more efficient, Martijn and I >> > >>>> volunteer >> > >>>>> to summarize the ongoing big things of different components and >> > >>> present a >> > >>>>> roadmap draft to the community in the next few weeks. This should >> be >> > >> a >> > >>>> good >> > >>>>> starting point for a more detailed discussion. >> > >>>>> >> > >>>>> Regarding the regular update mechanism, there was a proposal in a >> > >>> thread >> > >>>>> [2] three years ago to make the release manager responsible for >> > >>> updating >> > >>>>> the roadmap. However, it appears that this was not documented as a >> > >>>> release >> > >>>>> management task [3], and the roadmap update wasn't performed for >> > >>> releases >> > >>>>> 1.16 and 1.17. >> > >>>>> >> > >>>>> In my opinion, making release managers responsible for keeping the >> > >>>> roadmap >> > >>>>> up to date is a good idea. Specifically, release managers of >> release >> > >> X >> > >>>> can >> > >>>>> kick off the roadmap update at the beginning of release X, which >> can >> > >>> be a >> > >>>>> joint task with collecting a feature list [4]. Additionally, >> release >> > >>>>> managers of release X-1 can help verify and remove the >> accomplished >> > >>> items >> > >>>>> from the roadmap and update the feature radar. >> > >>>>> >> > >>>>> What do you think? Do you have other ideas? >> > >>>>> >> > >>>>> Best, >> > >>>>> Jark & Martijn >> > >>>>> >> > >>>>> [1]: https://flink.apache.org/roadmap.html >> > >>>>> [2]: >> > >> https://lists.apache.org/thread/o0l3cg6yphxwrww0k7215jgtw3yfoybv >> > >>>>> [3]: >> > >>>>> >> > >>>> >> > >>> >> > >> >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+Management >> > >>>>> [4]: >> https://cwiki.apache.org/confluence/display/FLINK/1.18+Release >> > >>>>> >> > >>>> >> > >>> >> > >> >> > >> > >> >
