FYI, release-3.3 branch has been cut and is ready for the release process for StateFun 3.3.0: https://github.com/apache/flink-statefun/tree/release-3.3
On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Btw, I'll assume that we're using this thread to gather consensus for > code-freezing for 3.3.x series of StateFun. I know there hasn't been much > activity on the repo, so this is just a formality really :) > > From the commit history, it looks like we're mainly including the below > major changes and bug fixes for 3.3.x: > - Flink upgrade to 1.15.2 > - https://issues.apache.org/jira/browse/FLINK-26340 > - https://issues.apache.org/jira/browse/FLINK-25866 > - https://issues.apache.org/jira/browse/FLINK-25936 > - https://issues.apache.org/jira/browse/FLINK-25933 > > I'll wait for 24 hours before cutting the release branch for 3.3.x, unless > anyone raises any objections before that. > > Thanks, > Gordon > > On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com> > wrote: > >> Thanks Gordon and Filip, I appreciate your help on this one. >> >> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts, >>> Galen and Filip! >>> >>> We should be ready to kick off a new release for StateFun with the Flink >>> version upgrade. >>> I'll cut off a release branch now on apache/flink-statefun for >>> release-3.3.x to move things forward. >>> @Galen, @Filip if you want to, after the release branch is cut, you could >>> probably upgrade the master branch to Flink 1.16.x as well. >>> >>> Afterwards we should decide who is available to drive the actual release >>> process for 3.3.0. >>> There's quite a few steps that would require committer write access. >>> Unless someone else is up for this earlier, I'll have some availability >>> towards the end of next week to help drive this. >>> >>> Thanks, >>> Gordon >>> >>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com> >>> wrote: >>> >>> > Yes, that makes sense. >>> > >>> > PR is here: [FLINK-29814][statefun] Change supported Flink version to >>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun >>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>. >>> > >>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <trohrm...@apache.org> >>> > wrote: >>> > >>> > > I think there might still be value in supporting 1.15 since not >>> everyone >>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between >>> Flink >>> > > 1.15 and 1.16 boils down to changing the Flink dependencies. >>> > > >>> > > Cheers, >>> > > Till >>> > > >>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren < >>> ga...@cvillewarrens.com> >>> > > wrote: >>> > > >>> > >> Sure thing. One question -- Flink 1.16 was just released a few days >>> ago. >>> > >> Should I support 1.15, or just go straight to 1.16? >>> > >> >>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrm...@apache.org >>> > >>> > >> wrote: >>> > >> >>> > >>> Hi folks, >>> > >>> >>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this >>> would >>> > be >>> > >>> awesome. I've assigned you to this ticket. The next thing after >>> merging >>> > >>> this PR would be creating a new StateFun release. Once we have >>> merged >>> > the >>> > >>> PR, let's check who can help with it the fastest. >>> > >>> >>> > >>> Cheers, >>> > >>> Till >>> > >>> >>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren < >>> ga...@cvillewarrens.com> >>> > >>> wrote: >>> > >>> >>> > >>>> Yes, I could do that. >>> > >>>> >>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki < >>> > >>>> filip.karni...@gmail.com> wrote: >>> > >>>> >>> > >>>>> Hi All >>> > >>>>> >>> > >>>>> So what's the play here? >>> > >>>>> >>> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till >>> would >>> > >>>>> assign this jira to you (with your permission) given he's helped >>> me >>> > out >>> > >>>>> with statefun work before >>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814 >>> > >>>>> >>> > >>>>> I can try to move to move statefun to flink 1.16 when it's out >>> > >>>>> >>> > >>>>> >>> > >>>>> Kind regards >>> > >>>>> Fil >>> > >>>>> >>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki < >>> > filip.karni...@gmail.com> >>> > >>>>> wrote: >>> > >>>>> >>> > >>>>>> Hi All >>> > >>>>>> >>> > >>>>>> Our use case is that we need to process elements for the same >>> key >>> > >>>>>> sequentially, and this processing involves async operations. >>> > >>>>>> >>> > >>>>>> If any part of the processing fails, we store the offending and >>> all >>> > >>>>>> subsequent incoming messages for that key in the state and not >>> > process any >>> > >>>>>> further messages for that key, until a retry succeeds or a human >>> > sends a >>> > >>>>>> 'skip' command message. >>> > >>>>>> >>> > >>>>>> diagram: >>> > >>>>>> >>> > >>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >>> > >>>>>> mermaid (in case mermaid.live goes down in the future): >>> > >>>>>> graph LR >>> > >>>>>> incoming[incoming events] --> job(Flink statefun job) >>> > >>>>>> commands[commands] -->|skip| job >>> > >>>>>> job --> |sequentially per key| remote(remote function) >>> > >>>>>> remote --> |on failure, delayed message to retry| remote >>> > >>>>>> remote --> |async puts/gets with side effects| other(other >>> > >>>>>> systems) >>> > >>>>>> >>> > >>>>>> Having the processing happen outside of Flink is nice-to-have >>> from >>> > an >>> > >>>>>> independent scalability point of view, but is not strictly >>> required. >>> > >>>>>> >>> > >>>>>> So long story short - no cyclic messaging, but also no way I can >>> > >>>>>> think of to use existing native Flink operators like async i/o >>> > (which when >>> > >>>>>> I last checked a few years back didn't have access to keyed >>> state) >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> P.S. Please note that there is already a pull request that has >>> > >>>>>> something to do wtih Flink 1.15, albeit without a description >>> or a >>> > jira: >>> > >>>>>> https://github.com/apache/flink-statefun/pull/314 >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren < >>> ga...@cvillewarrens.com >>> > > >>> > >>>>>> wrote: >>> > >>>>>> >>> > >>>>>>> Hi Gordon (and others), >>> > >>>>>>> >>> > >>>>>>> I'm also using this project for stateful messaging, including >>> > >>>>>>> messaging >>> > >>>>>>> among functions. >>> > >>>>>>> >>> > >>>>>>> I've contributed a small amount of code in the past and have >>> also >>> > >>>>>>> enabled >>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to >>> > help >>> > >>>>>>> out >>> > >>>>>>> here. >>> > >>>>>>> >>> > >>>>>>> Thanks, >>> > >>>>>>> Galen >>> > >>>>>>> >>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < >>> > >>>>>>> kkrugler_li...@transpac.com> >>> > >>>>>>> wrote: >>> > >>>>>>> >>> > >>>>>>> > Hi Gordon, >>> > >>>>>>> > >>> > >>>>>>> > We’re using it for stateful messaging, and also calling >>> remote >>> > >>>>>>> > Python-based functions. >>> > >>>>>>> > >>> > >>>>>>> > So yes, also very interested in what is going to happen with >>> the >>> > >>>>>>> this >>> > >>>>>>> > subproject in the future. >>> > >>>>>>> > >>> > >>>>>>> > — Ken >>> > >>>>>>> > >>> > >>>>>>> > >>> > >>>>>>> > >>> > >>>>>>> > > Begin forwarded message: >>> > >>>>>>> > > >>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT >>> > >>>>>>> > > To: dev@flink.apache.org >>> > >>>>>>> > > Reply-To: dev@flink.apache.org >>> > >>>>>>> > > >>> > >>>>>>> > > Hi Filip, >>> > >>>>>>> > > >>> > >>>>>>> > > Thanks for bringing this up. >>> > >>>>>>> > > >>> > >>>>>>> > > The hard truth is that committers who were previously >>> active on >>> > >>>>>>> the >>> > >>>>>>> > > StateFun subproject, including myself, all currently have >>> other >>> > >>>>>>> focuses. >>> > >>>>>>> > > Indeed, we may need to discuss with the community on how to >>> > >>>>>>> proceed if >>> > >>>>>>> > > there seems to be no continued committer coverage. >>> > >>>>>>> > > >>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink >>> version, >>> > >>>>>>> I'm still >>> > >>>>>>> > > familiar enough with the subproject to probably be able to >>> > drive >>> > >>>>>>> this (or >>> > >>>>>>> > > if your team is up to it, I can assist you on that). >>> > >>>>>>> > > >>> > >>>>>>> > > For the long-term, as a data point I'm curious to see how >>> many >>> > >>>>>>> users are >>> > >>>>>>> > > using StateFun in production today, and how you're using >>> it? >>> > >>>>>>> > > >>> > >>>>>>> > > - Do your applications have arbitrary / cyclic / >>> > bi-directional >>> > >>>>>>> > > messaging between individual functions? >>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your >>> stateful >>> > >>>>>>> functions >>> > >>>>>>> > > to run remotely as separate processes? >>> > >>>>>>> > > >>> > >>>>>>> > > If the majority is only the latter category, there might >>> be a >>> > >>>>>>> case to >>> > >>>>>>> > > support remote functions natively in Flink (which has been >>> a >>> > >>>>>>> discussion >>> > >>>>>>> > in >>> > >>>>>>> > > the past). >>> > >>>>>>> > > >>> > >>>>>>> > > Thanks, >>> > >>>>>>> > > Gordon >>> > >>>>>>> > > >>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >>> > >>>>>>> filip.karni...@gmail.com >>> > >>>>>>> > > >>> > >>>>>>> > > wrote: >>> > >>>>>>> > > >>> > >>>>>>> > >> Hi, I noticed that the development on stateful functions >>> has >>> > >>>>>>> come to a >>> > >>>>>>> > bit >>> > >>>>>>> > >> of a halt, with a pull request to update statefun to use >>> Flink >>> > >>>>>>> 1.15 >>> > >>>>>>> > being >>> > >>>>>>> > >> in the `open` state since May 2022. >>> > >>>>>>> > >> >>> > >>>>>>> > >> What do we think is the future of this sub-project? >>> > >>>>>>> > >> >>> > >>>>>>> > >> The background to this question is that my team is on a >>> shared >>> > >>>>>>> Flink >>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I >>> need >>> > to >>> > >>>>>>> > re-write >>> > >>>>>>> > >> all our code as a native Flink job (rather than a remote >>> > >>>>>>> stateful >>> > >>>>>>> > function) >>> > >>>>>>> > >> then I need to get started right away. >>> > >>>>>>> > >> >>> > >>>>>>> > >> Many thanks, >>> > >>>>>>> > >> Fil >>> > >>>>>>> > >> >>> > >>>>>>> > >>> > >>>>>>> > -------------------------- >>> > >>>>>>> > Ken Krugler >>> > >>>>>>> > http://www.scaleunlimited.com >>> > >>>>>>> > Custom big data solutions >>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch >>> > >>>>>>> > >>> > >>>>>>> > >>> > >>>>>>> > >>> > >>>>>>> > >>> > >>>>>>> >>> > >>>>>> >>> > >>> >>
