Thanks Gordon. What is the next step here?
On Thu, Nov 3, 2022 at 1:45 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > FYI, release-3.3 branch has been cut and is ready for the release process > for StateFun 3.3.0: > https://github.com/apache/flink-statefun/tree/release-3.3 > > On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Btw, I'll assume that we're using this thread to gather consensus for >> code-freezing for 3.3.x series of StateFun. I know there hasn't been much >> activity on the repo, so this is just a formality really :) >> >> From the commit history, it looks like we're mainly including the below >> major changes and bug fixes for 3.3.x: >> - Flink upgrade to 1.15.2 >> - https://issues.apache.org/jira/browse/FLINK-26340 >> - https://issues.apache.org/jira/browse/FLINK-25866 >> - https://issues.apache.org/jira/browse/FLINK-25936 >> - https://issues.apache.org/jira/browse/FLINK-25933 >> >> I'll wait for 24 hours before cutting the release branch for 3.3.x, >> unless anyone raises any objections before that. >> >> Thanks, >> Gordon >> >> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com> >> wrote: >> >>> Thanks Gordon and Filip, I appreciate your help on this one. >>> >>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> >>> wrote: >>> >>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the >>>> efforts, >>>> Galen and Filip! >>>> >>>> We should be ready to kick off a new release for StateFun with the Flink >>>> version upgrade. >>>> I'll cut off a release branch now on apache/flink-statefun for >>>> release-3.3.x to move things forward. >>>> @Galen, @Filip if you want to, after the release branch is cut, you >>>> could >>>> probably upgrade the master branch to Flink 1.16.x as well. >>>> >>>> Afterwards we should decide who is available to drive the actual release >>>> process for 3.3.0. >>>> There's quite a few steps that would require committer write access. >>>> Unless someone else is up for this earlier, I'll have some availability >>>> towards the end of next week to help drive this. >>>> >>>> Thanks, >>>> Gordon >>>> >>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com> >>>> wrote: >>>> >>>> > Yes, that makes sense. >>>> > >>>> > PR is here: [FLINK-29814][statefun] Change supported Flink version to >>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun >>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319>. >>>> > >>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <trohrm...@apache.org> >>>> > wrote: >>>> > >>>> > > I think there might still be value in supporting 1.15 since not >>>> everyone >>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff between >>>> Flink >>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies. >>>> > > >>>> > > Cheers, >>>> > > Till >>>> > > >>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren < >>>> ga...@cvillewarrens.com> >>>> > > wrote: >>>> > > >>>> > >> Sure thing. One question -- Flink 1.16 was just released a few >>>> days ago. >>>> > >> Should I support 1.15, or just go straight to 1.16? >>>> > >> >>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann < >>>> trohrm...@apache.org> >>>> > >> wrote: >>>> > >> >>>> > >>> Hi folks, >>>> > >>> >>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this >>>> would >>>> > be >>>> > >>> awesome. I've assigned you to this ticket. The next thing after >>>> merging >>>> > >>> this PR would be creating a new StateFun release. Once we have >>>> merged >>>> > the >>>> > >>> PR, let's check who can help with it the fastest. >>>> > >>> >>>> > >>> Cheers, >>>> > >>> Till >>>> > >>> >>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren < >>>> ga...@cvillewarrens.com> >>>> > >>> wrote: >>>> > >>> >>>> > >>>> Yes, I could do that. >>>> > >>>> >>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki < >>>> > >>>> filip.karni...@gmail.com> wrote: >>>> > >>>> >>>> > >>>>> Hi All >>>> > >>>>> >>>> > >>>>> So what's the play here? >>>> > >>>>> >>>> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till >>>> would >>>> > >>>>> assign this jira to you (with your permission) given he's >>>> helped me >>>> > out >>>> > >>>>> with statefun work before >>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814 >>>> > >>>>> >>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's out >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> Kind regards >>>> > >>>>> Fil >>>> > >>>>> >>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki < >>>> > filip.karni...@gmail.com> >>>> > >>>>> wrote: >>>> > >>>>> >>>> > >>>>>> Hi All >>>> > >>>>>> >>>> > >>>>>> Our use case is that we need to process elements for the same >>>> key >>>> > >>>>>> sequentially, and this processing involves async operations. >>>> > >>>>>> >>>> > >>>>>> If any part of the processing fails, we store the offending >>>> and all >>>> > >>>>>> subsequent incoming messages for that key in the state and not >>>> > process any >>>> > >>>>>> further messages for that key, until a retry succeeds or a >>>> human >>>> > sends a >>>> > >>>>>> 'skip' command message. >>>> > >>>>>> >>>> > >>>>>> diagram: >>>> > >>>>>> >>>> > >>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >>>> > >>>>>> mermaid (in case mermaid.live goes down in the future): >>>> > >>>>>> graph LR >>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job) >>>> > >>>>>> commands[commands] -->|skip| job >>>> > >>>>>> job --> |sequentially per key| remote(remote function) >>>> > >>>>>> remote --> |on failure, delayed message to retry| remote >>>> > >>>>>> remote --> |async puts/gets with side effects| other(other >>>> > >>>>>> systems) >>>> > >>>>>> >>>> > >>>>>> Having the processing happen outside of Flink is nice-to-have >>>> from >>>> > an >>>> > >>>>>> independent scalability point of view, but is not strictly >>>> required. >>>> > >>>>>> >>>> > >>>>>> So long story short - no cyclic messaging, but also no way I >>>> can >>>> > >>>>>> think of to use existing native Flink operators like async i/o >>>> > (which when >>>> > >>>>>> I last checked a few years back didn't have access to keyed >>>> state) >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> P.S. Please note that there is already a pull request that has >>>> > >>>>>> something to do wtih Flink 1.15, albeit without a description >>>> or a >>>> > jira: >>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314 >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren < >>>> ga...@cvillewarrens.com >>>> > > >>>> > >>>>>> wrote: >>>> > >>>>>> >>>> > >>>>>>> Hi Gordon (and others), >>>> > >>>>>>> >>>> > >>>>>>> I'm also using this project for stateful messaging, including >>>> > >>>>>>> messaging >>>> > >>>>>>> among functions. >>>> > >>>>>>> >>>> > >>>>>>> I've contributed a small amount of code in the past and have >>>> also >>>> > >>>>>>> enabled >>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able >>>> to >>>> > help >>>> > >>>>>>> out >>>> > >>>>>>> here. >>>> > >>>>>>> >>>> > >>>>>>> Thanks, >>>> > >>>>>>> Galen >>>> > >>>>>>> >>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < >>>> > >>>>>>> kkrugler_li...@transpac.com> >>>> > >>>>>>> wrote: >>>> > >>>>>>> >>>> > >>>>>>> > Hi Gordon, >>>> > >>>>>>> > >>>> > >>>>>>> > We’re using it for stateful messaging, and also calling >>>> remote >>>> > >>>>>>> > Python-based functions. >>>> > >>>>>>> > >>>> > >>>>>>> > So yes, also very interested in what is going to happen >>>> with the >>>> > >>>>>>> this >>>> > >>>>>>> > subproject in the future. >>>> > >>>>>>> > >>>> > >>>>>>> > — Ken >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> > > Begin forwarded message: >>>> > >>>>>>> > > >>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and >>>> onwards >>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT >>>> > >>>>>>> > > To: dev@flink.apache.org >>>> > >>>>>>> > > Reply-To: dev@flink.apache.org >>>> > >>>>>>> > > >>>> > >>>>>>> > > Hi Filip, >>>> > >>>>>>> > > >>>> > >>>>>>> > > Thanks for bringing this up. >>>> > >>>>>>> > > >>>> > >>>>>>> > > The hard truth is that committers who were previously >>>> active on >>>> > >>>>>>> the >>>> > >>>>>>> > > StateFun subproject, including myself, all currently have >>>> other >>>> > >>>>>>> focuses. >>>> > >>>>>>> > > Indeed, we may need to discuss with the community on how >>>> to >>>> > >>>>>>> proceed if >>>> > >>>>>>> > > there seems to be no continued committer coverage. >>>> > >>>>>>> > > >>>> > >>>>>>> > > If it's just a matter of upgrading the supported Flink >>>> version, >>>> > >>>>>>> I'm still >>>> > >>>>>>> > > familiar enough with the subproject to probably be able to >>>> > drive >>>> > >>>>>>> this (or >>>> > >>>>>>> > > if your team is up to it, I can assist you on that). >>>> > >>>>>>> > > >>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see how >>>> many >>>> > >>>>>>> users are >>>> > >>>>>>> > > using StateFun in production today, and how you're using >>>> it? >>>> > >>>>>>> > > >>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic / >>>> > bi-directional >>>> > >>>>>>> > > messaging between individual functions? >>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your >>>> stateful >>>> > >>>>>>> functions >>>> > >>>>>>> > > to run remotely as separate processes? >>>> > >>>>>>> > > >>>> > >>>>>>> > > If the majority is only the latter category, there might >>>> be a >>>> > >>>>>>> case to >>>> > >>>>>>> > > support remote functions natively in Flink (which has >>>> been a >>>> > >>>>>>> discussion >>>> > >>>>>>> > in >>>> > >>>>>>> > > the past). >>>> > >>>>>>> > > >>>> > >>>>>>> > > Thanks, >>>> > >>>>>>> > > Gordon >>>> > >>>>>>> > > >>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >>>> > >>>>>>> filip.karni...@gmail.com >>>> > >>>>>>> > > >>>> > >>>>>>> > > wrote: >>>> > >>>>>>> > > >>>> > >>>>>>> > >> Hi, I noticed that the development on stateful functions >>>> has >>>> > >>>>>>> come to a >>>> > >>>>>>> > bit >>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to use >>>> Flink >>>> > >>>>>>> 1.15 >>>> > >>>>>>> > being >>>> > >>>>>>> > >> in the `open` state since May 2022. >>>> > >>>>>>> > >> >>>> > >>>>>>> > >> What do we think is the future of this sub-project? >>>> > >>>>>>> > >> >>>> > >>>>>>> > >> The background to this question is that my team is on a >>>> shared >>>> > >>>>>>> Flink >>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I >>>> need >>>> > to >>>> > >>>>>>> > re-write >>>> > >>>>>>> > >> all our code as a native Flink job (rather than a remote >>>> > >>>>>>> stateful >>>> > >>>>>>> > function) >>>> > >>>>>>> > >> then I need to get started right away. >>>> > >>>>>>> > >> >>>> > >>>>>>> > >> Many thanks, >>>> > >>>>>>> > >> Fil >>>> > >>>>>>> > >> >>>> > >>>>>>> > >>>> > >>>>>>> > -------------------------- >>>> > >>>>>>> > Ken Krugler >>>> > >>>>>>> > http://www.scaleunlimited.com >>>> > >>>>>>> > Custom big data solutions >>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> >>>> > >>>>>> >>>> > >>>> >>>
