Hello! Is there any update on this thread? I’m happy to help in any way I can too.
On 2023/01/19 16:04:53 Galen Warren wrote: > I thought I'd check in again here, is there any update? If there's anything > I can do to help with releasing statefun 3.3, just let me know. Thanks. > > On Mon, Nov 28, 2022 at 8:12 AM Galen Warren <ga...@cvillewarrens.com> > wrote: > > > Hi Gordon, just checking in. Is there anything I can do to help here? > > > > On Fri, Nov 4, 2022 at 12:59 PM Tzu-Li (Gordon) Tai <tz...@apache.org> > > wrote: > > > >> @Galen The next step is essentially for someone to be the release manager > >> and drive the release process for StateFun 3.3.0 [1]. The document for the > >> release process might be slightly outdated in some places, but overall > >> outlines the process pretty clearly. > >> > >> As I mentioned earlier, there's quite a few steps in the process that > >> requires committer write access, so it would be easier if a committer can > >> pick this up as the release manager. > >> I'll be happy to take this on, but I'll only have availability after next > >> week. If someone else is willing to take this on earlier it would > >> certainly > >> be better to get the release out ASAP. > >> > >> [1] > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Stateful+Functions+Release > >> > >> On Fri, Nov 4, 2022 at 5:54 AM Galen Warren <ga...@cvillewarrens.com> > >> wrote: > >> > >> > Thanks Gordon. > >> > > >> > What is the next step here? > >> > > >> > On Thu, Nov 3, 2022 at 1:45 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org > >> > > >> > wrote: > >> > > >> > > FYI, release-3.3 branch has been cut and is ready for the release > >> process > >> > > for StateFun 3.3.0: > >> > > https://github.com/apache/flink-statefun/tree/release-3.3 > >> > > > >> > > On Tue, Nov 1, 2022 at 10:21 AM Tzu-Li (Gordon) Tai < > >> tzuli...@apache.org > >> > > > >> > > wrote: > >> > > > >> > >> Btw, I'll assume that we're using this thread to gather consensus for > >> > >> code-freezing for 3.3.x series of StateFun. I know there hasn't been > >> > much > >> > >> activity on the repo, so this is just a formality really :) > >> > >> > >> > >> From the commit history, it looks like we're mainly including the > >> below > >> > >> major changes and bug fixes for 3.3.x: > >> > >> - Flink upgrade to 1.15.2 > >> > >> - https://issues.apache.org/jira/browse/FLINK-26340 > >> > >> - https://issues.apache.org/jira/browse/FLINK-25866 > >> > >> - https://issues.apache.org/jira/browse/FLINK-25936 > >> > >> - https://issues.apache.org/jira/browse/FLINK-25933 > >> > >> > >> > >> I'll wait for 24 hours before cutting the release branch for 3.3.x, > >> > >> unless anyone raises any objections before that. > >> > >> > >> > >> Thanks, > >> > >> Gordon > >> > >> > >> > >> On Tue, Nov 1, 2022 at 10:09 AM Galen Warren < > >> ga...@cvillewarrens.com> > >> > >> wrote: > >> > >> > >> > >>> Thanks Gordon and Filip, I appreciate your help on this one. > >> > >>> > >> > >>> On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai < > >> > tzuli...@apache.org> > >> > >>> wrote: > >> > >>> > >> > >>>> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the > >> > >>>> efforts, > >> > >>>> Galen and Filip! > >> > >>>> > >> > >>>> We should be ready to kick off a new release for StateFun with the > >> > Flink > >> > >>>> version upgrade. > >> > >>>> I'll cut off a release branch now on apache/flink-statefun for > >> > >>>> release-3.3.x to move things forward. > >> > >>>> @Galen, @Filip if you want to, after the release branch is cut, you > >> > >>>> could > >> > >>>> probably upgrade the master branch to Flink 1.16.x as well. > >> > >>>> > >> > >>>> Afterwards we should decide who is available to drive the actual > >> > release > >> > >>>> process for 3.3.0. > >> > >>>> There's quite a few steps that would require committer write > >> access. > >> > >>>> Unless someone else is up for this earlier, I'll have some > >> > availability > >> > >>>> towards the end of next week to help drive this. > >> > >>>> > >> > >>>> Thanks, > >> > >>>> Gordon > >> > >>>> > >> > >>>> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren < > >> > ga...@cvillewarrens.com> > >> > >>>> wrote: > >> > >>>> > >> > >>>> > Yes, that makes sense. > >> > >>>> > > >> > >>>> > PR is here: [FLINK-29814][statefun] Change supported Flink > >> version > >> > to > >> > >>>> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun > >> > >>>> > (github.com) <https://github.com/apache/flink-statefun/pull/319 > >> >. > >> > >>>> > > >> > >>>> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann < > >> > trohrm...@apache.org> > >> > >>>> > wrote: > >> > >>>> > > >> > >>>> > > I think there might still be value in supporting 1.15 since not > >> > >>>> everyone > >> > >>>> > > upgrades Flink very fast. Hopefully, for Statefun the diff > >> between > >> > >>>> Flink > >> > >>>> > > 1.15 and 1.16 boils down to changing the Flink dependencies. > >> > >>>> > > > >> > >>>> > > Cheers, > >> > >>>> > > Till > >> > >>>> > > > >> > >>>> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren < > >> > >>>> ga...@cvillewarrens.com> > >> > >>>> > > wrote: > >> > >>>> > > > >> > >>>> > >> Sure thing. One question -- Flink 1.16 was just released a few > >> > >>>> days ago. > >> > >>>> > >> Should I support 1.15, or just go straight to 1.16? > >> > >>>> > >> > >> > >>>> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann < > >> > >>>> trohrm...@apache.org> > >> > >>>> > >> wrote: > >> > >>>> > >> > >> > >>>> > >>> Hi folks, > >> > >>>> > >>> > >> > >>>> > >>> if you can open a PR for supporting Flink 1.15 Galen, then > >> this > >> > >>>> would > >> > >>>> > be > >> > >>>> > >>> awesome. I've assigned you to this ticket. The next thing > >> after > >> > >>>> merging > >> > >>>> > >>> this PR would be creating a new StateFun release. Once we > >> have > >> > >>>> merged > >> > >>>> > the > >> > >>>> > >>> PR, let's check who can help with it the fastest. > >> > >>>> > >>> > >> > >>>> > >>> Cheers, > >> > >>>> > >>> Till > >> > >>>> > >>> > >> > >>>> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren < > >> > >>>> ga...@cvillewarrens.com> > >> > >>>> > >>> wrote: > >> > >>>> > >>> > >> > >>>> > >>>> Yes, I could do that. > >> > >>>> > >>>> > >> > >>>> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki < > >> > >>>> > >>>> filip.karni...@gmail.com> wrote: > >> > >>>> > >>>> > >> > >>>> > >>>>> Hi All > >> > >>>> > >>>>> > >> > >>>> > >>>>> So what's the play here? > >> > >>>> > >>>>> > >> > >>>> > >>>>> Galen, what do you think about taking this on? Perhaps > >> ++Till > >> > >>>> would > >> > >>>> > >>>>> assign this jira to you (with your permission) given he's > >> > >>>> helped me > >> > >>>> > out > >> > >>>> > >>>>> with statefun work before > >> > >>>> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814 > >> > >>>> > >>>>> > >> > >>>> > >>>>> I can try to move to move statefun to flink 1.16 when it's > >> out > >> > >>>> > >>>>> > >> > >>>> > >>>>> > >> > >>>> > >>>>> Kind regards > >> > >>>> > >>>>> Fil > >> > >>>> > >>>>> > >> > >>>> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki < > >> > >>>> > filip.karni...@gmail.com> > >> > >>>> > >>>>> wrote: > >> > >>>> > >>>>> > >> > >>>> > >>>>>> Hi All > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> Our use case is that we need to process elements for the > >> same > >> > >>>> key > >> > >>>> > >>>>>> sequentially, and this processing involves async > >> operations. > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> If any part of the processing fails, we store the > >> offending > >> > >>>> and all > >> > >>>> > >>>>>> subsequent incoming messages for that key in the state and > >> > not > >> > >>>> > process any > >> > >>>> > >>>>>> further messages for that key, until a retry succeeds or a > >> > >>>> human > >> > >>>> > sends a > >> > >>>> > >>>>>> 'skip' command message. > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> diagram: > >> > >>>> > >>>>>> > >> > >>>> > > >> > >>>> > >> > > >> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek > >> > >>>> > >>>>>> mermaid (in case mermaid.live goes down in the future): > >> > >>>> > >>>>>> graph LR > >> > >>>> > >>>>>> incoming[incoming events] --> job(Flink statefun job) > >> > >>>> > >>>>>> commands[commands] -->|skip| job > >> > >>>> > >>>>>> job --> |sequentially per key| remote(remote function) > >> > >>>> > >>>>>> remote --> |on failure, delayed message to retry| > >> remote > >> > >>>> > >>>>>> remote --> |async puts/gets with side effects| > >> > other(other > >> > >>>> > >>>>>> systems) > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> Having the processing happen outside of Flink is > >> nice-to-have > >> > >>>> from > >> > >>>> > an > >> > >>>> > >>>>>> independent scalability point of view, but is not strictly > >> > >>>> required. > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> So long story short - no cyclic messaging, but also no > >> way I > >> > >>>> can > >> > >>>> > >>>>>> think of to use existing native Flink operators like async > >> > i/o > >> > >>>> > (which when > >> > >>>> > >>>>>> I last checked a few years back didn't have access to > >> keyed > >> > >>>> state) > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> P.S. Please note that there is already a pull request that > >> > has > >> > >>>> > >>>>>> something to do wtih Flink 1.15, albeit without a > >> description > >> > >>>> or a > >> > >>>> > jira: > >> > >>>> > >>>>>> https://github.com/apache/flink-statefun/pull/314 > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren < > >> > >>>> ga...@cvillewarrens.com > >> > >>>> > > > >> > >>>> > >>>>>> wrote: > >> > >>>> > >>>>>> > >> > >>>> > >>>>>>> Hi Gordon (and others), > >> > >>>> > >>>>>>> > >> > >>>> > >>>>>>> I'm also using this project for stateful messaging, > >> > including > >> > >>>> > >>>>>>> messaging > >> > >>>> > >>>>>>> among functions. > >> > >>>> > >>>>>>> > >> > >>>> > >>>>>>> I've contributed a small amount of code in the past and > >> have > >> > >>>> also > >> > >>>> > >>>>>>> enabled > >> > >>>> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be > >> able > >> > >>>> to > >> > >>>> > help > >> > >>>> > >>>>>>> out > >> > >>>> > >>>>>>> here. > >> > >>>> > >>>>>>> > >> > >>>> > >>>>>>> Thanks, > >> > >>>> > >>>>>>> Galen > >> > >>>> > >>>>>>> > >> > >>>> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < > >> > >>>> > >>>>>>> kkrugler_li...@transpac.com> > >> > >>>> > >>>>>>> wrote: > >> > >>>> > >>>>>>> > >> > >>>> > >>>>>>> > Hi Gordon, > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > We’re using it for stateful messaging, and also calling > >> > >>>> remote > >> > >>>> > >>>>>>> > Python-based functions. > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > So yes, also very interested in what is going to happen > >> > >>>> with the > >> > >>>> > >>>>>>> this > >> > >>>> > >>>>>>> > subproject in the future. > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > — Ken > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > > Begin forwarded message: > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tz...@apache.org> > >> > >>>> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and > >> > >>>> onwards > >> > >>>> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT > >> > >>>> > >>>>>>> > > To: dev@flink.apache.org > >> > >>>> > >>>>>>> > > Reply-To: dev@flink.apache.org > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > Hi Filip, > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > Thanks for bringing this up. > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > The hard truth is that committers who were previously > >> > >>>> active on > >> > >>>> > >>>>>>> the > >> > >>>> > >>>>>>> > > StateFun subproject, including myself, all currently > >> > have > >> > >>>> other > >> > >>>> > >>>>>>> focuses. > >> > >>>> > >>>>>>> > > Indeed, we may need to discuss with the community on > >> how > >> > >>>> to > >> > >>>> > >>>>>>> proceed if > >> > >>>> > >>>>>>> > > there seems to be no continued committer coverage. > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > If it's just a matter of upgrading the supported > >> Flink > >> > >>>> version, > >> > >>>> > >>>>>>> I'm still > >> > >>>> > >>>>>>> > > familiar enough with the subproject to probably be > >> able > >> > to > >> > >>>> > drive > >> > >>>> > >>>>>>> this (or > >> > >>>> > >>>>>>> > > if your team is up to it, I can assist you on that). > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > For the long-term, as a data point I'm curious to see > >> > how > >> > >>>> many > >> > >>>> > >>>>>>> users are > >> > >>>> > >>>>>>> > > using StateFun in production today, and how you're > >> using > >> > >>>> it? > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > - Do your applications have arbitrary / cyclic / > >> > >>>> > bi-directional > >> > >>>> > >>>>>>> > > messaging between individual functions? > >> > >>>> > >>>>>>> > > - Or are you utilizing StateFun simply to allow > >> your > >> > >>>> stateful > >> > >>>> > >>>>>>> functions > >> > >>>> > >>>>>>> > > to run remotely as separate processes? > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > If the majority is only the latter category, there > >> might > >> > >>>> be a > >> > >>>> > >>>>>>> case to > >> > >>>> > >>>>>>> > > support remote functions natively in Flink (which has > >> > >>>> been a > >> > >>>> > >>>>>>> discussion > >> > >>>> > >>>>>>> > in > >> > >>>> > >>>>>>> > > the past). > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > Thanks, > >> > >>>> > >>>>>>> > > Gordon > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < > >> > >>>> > >>>>>>> filip.karni...@gmail.com > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > > wrote: > >> > >>>> > >>>>>>> > > > >> > >>>> > >>>>>>> > >> Hi, I noticed that the development on stateful > >> > functions > >> > >>>> has > >> > >>>> > >>>>>>> come to a > >> > >>>> > >>>>>>> > bit > >> > >>>> > >>>>>>> > >> of a halt, with a pull request to update statefun to > >> > use > >> > >>>> Flink > >> > >>>> > >>>>>>> 1.15 > >> > >>>> > >>>>>>> > being > >> > >>>> > >>>>>>> > >> in the `open` state since May 2022. > >> > >>>> > >>>>>>> > >> > >> > >>>> > >>>>>>> > >> What do we think is the future of this sub-project? > >> > >>>> > >>>>>>> > >> > >> > >>>> > >>>>>>> > >> The background to this question is that my team is > >> on a > >> > >>>> shared > >> > >>>> > >>>>>>> Flink > >> > >>>> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. > >> If > >> > I > >> > >>>> need > >> > >>>> > to > >> > >>>> > >>>>>>> > re-write > >> > >>>> > >>>>>>> > >> all our code as a native Flink job (rather than a > >> > remote > >> > >>>> > >>>>>>> stateful > >> > >>>> > >>>>>>> > function) > >> > >>>> > >>>>>>> > >> then I need to get started right away. > >> > >>>> > >>>>>>> > >> > >> > >>>> > >>>>>>> > >> Many thanks, > >> > >>>> > >>>>>>> > >> Fil > >> > >>>> > >>>>>>> > >> > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > -------------------------- > >> > >>>> > >>>>>>> > Ken Krugler > >> > >>>> > >>>>>>> > http://www.scaleunlimited.com > >> > >>>> > >>>>>>> > Custom big data solutions > >> > >>>> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > > >> > >>>> > >>>>>>> > >> > >>>> > >>>>>> > >> > >>>> > > >> > >>>> > >> > >>> > >> > > >> > > > Dheeraj Balakavi Software Engineer - Pipeline Group dbalakavi <mailto:dbalak...@pagerduty.com>@pagerduty.com <mailto:dbalak...@pagerduty.com> Pronouns: he/him
