Yes, that makes sense. PR is here: [FLINK-29814][statefun] Change supported Flink version to 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun (github.com) <https://github.com/apache/flink-statefun/pull/319>.
On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <trohrm...@apache.org> wrote: > I think there might still be value in supporting 1.15 since not everyone > upgrades Flink very fast. Hopefully, for Statefun the diff between Flink > 1.15 and 1.16 boils down to changing the Flink dependencies. > > Cheers, > Till > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com> > wrote: > >> Sure thing. One question -- Flink 1.16 was just released a few days ago. >> Should I support 1.15, or just go straight to 1.16? >> >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >>> Hi folks, >>> >>> if you can open a PR for supporting Flink 1.15 Galen, then this would be >>> awesome. I've assigned you to this ticket. The next thing after merging >>> this PR would be creating a new StateFun release. Once we have merged the >>> PR, let's check who can help with it the fastest. >>> >>> Cheers, >>> Till >>> >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com> >>> wrote: >>> >>>> Yes, I could do that. >>>> >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki < >>>> filip.karni...@gmail.com> wrote: >>>> >>>>> Hi All >>>>> >>>>> So what's the play here? >>>>> >>>>> Galen, what do you think about taking this on? Perhaps ++Till would >>>>> assign this jira to you (with your permission) given he's helped me out >>>>> with statefun work before >>>>> https://issues.apache.org/jira/browse/FLINK-29814 >>>>> >>>>> I can try to move to move statefun to flink 1.16 when it's out >>>>> >>>>> >>>>> Kind regards >>>>> Fil >>>>> >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <filip.karni...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi All >>>>>> >>>>>> Our use case is that we need to process elements for the same key >>>>>> sequentially, and this processing involves async operations. >>>>>> >>>>>> If any part of the processing fails, we store the offending and all >>>>>> subsequent incoming messages for that key in the state and not process >>>>>> any >>>>>> further messages for that key, until a retry succeeds or a human sends a >>>>>> 'skip' command message. >>>>>> >>>>>> diagram: >>>>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >>>>>> mermaid (in case mermaid.live goes down in the future): >>>>>> graph LR >>>>>> incoming[incoming events] --> job(Flink statefun job) >>>>>> commands[commands] -->|skip| job >>>>>> job --> |sequentially per key| remote(remote function) >>>>>> remote --> |on failure, delayed message to retry| remote >>>>>> remote --> |async puts/gets with side effects| other(other >>>>>> systems) >>>>>> >>>>>> Having the processing happen outside of Flink is nice-to-have from an >>>>>> independent scalability point of view, but is not strictly required. >>>>>> >>>>>> So long story short - no cyclic messaging, but also no way I can >>>>>> think of to use existing native Flink operators like async i/o (which >>>>>> when >>>>>> I last checked a few years back didn't have access to keyed state) >>>>>> >>>>>> >>>>>> P.S. Please note that there is already a pull request that has >>>>>> something to do wtih Flink 1.15, albeit without a description or a jira: >>>>>> https://github.com/apache/flink-statefun/pull/314 >>>>>> >>>>>> >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Gordon (and others), >>>>>>> >>>>>>> I'm also using this project for stateful messaging, including >>>>>>> messaging >>>>>>> among functions. >>>>>>> >>>>>>> I've contributed a small amount of code in the past and have also >>>>>>> enabled >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to help >>>>>>> out >>>>>>> here. >>>>>>> >>>>>>> Thanks, >>>>>>> Galen >>>>>>> >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < >>>>>>> kkrugler_li...@transpac.com> >>>>>>> wrote: >>>>>>> >>>>>>> > Hi Gordon, >>>>>>> > >>>>>>> > We’re using it for stateful messaging, and also calling remote >>>>>>> > Python-based functions. >>>>>>> > >>>>>>> > So yes, also very interested in what is going to happen with the >>>>>>> this >>>>>>> > subproject in the future. >>>>>>> > >>>>>>> > — Ken >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > > Begin forwarded message: >>>>>>> > > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT >>>>>>> > > To: dev@flink.apache.org >>>>>>> > > Reply-To: dev@flink.apache.org >>>>>>> > > >>>>>>> > > Hi Filip, >>>>>>> > > >>>>>>> > > Thanks for bringing this up. >>>>>>> > > >>>>>>> > > The hard truth is that committers who were previously active on >>>>>>> the >>>>>>> > > StateFun subproject, including myself, all currently have other >>>>>>> focuses. >>>>>>> > > Indeed, we may need to discuss with the community on how to >>>>>>> proceed if >>>>>>> > > there seems to be no continued committer coverage. >>>>>>> > > >>>>>>> > > If it's just a matter of upgrading the supported Flink version, >>>>>>> I'm still >>>>>>> > > familiar enough with the subproject to probably be able to drive >>>>>>> this (or >>>>>>> > > if your team is up to it, I can assist you on that). >>>>>>> > > >>>>>>> > > For the long-term, as a data point I'm curious to see how many >>>>>>> users are >>>>>>> > > using StateFun in production today, and how you're using it? >>>>>>> > > >>>>>>> > > - Do your applications have arbitrary / cyclic / bi-directional >>>>>>> > > messaging between individual functions? >>>>>>> > > - Or are you utilizing StateFun simply to allow your stateful >>>>>>> functions >>>>>>> > > to run remotely as separate processes? >>>>>>> > > >>>>>>> > > If the majority is only the latter category, there might be a >>>>>>> case to >>>>>>> > > support remote functions natively in Flink (which has been a >>>>>>> discussion >>>>>>> > in >>>>>>> > > the past). >>>>>>> > > >>>>>>> > > Thanks, >>>>>>> > > Gordon >>>>>>> > > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >>>>>>> filip.karni...@gmail.com >>>>>>> > > >>>>>>> > > wrote: >>>>>>> > > >>>>>>> > >> Hi, I noticed that the development on stateful functions has >>>>>>> come to a >>>>>>> > bit >>>>>>> > >> of a halt, with a pull request to update statefun to use Flink >>>>>>> 1.15 >>>>>>> > being >>>>>>> > >> in the `open` state since May 2022. >>>>>>> > >> >>>>>>> > >> What do we think is the future of this sub-project? >>>>>>> > >> >>>>>>> > >> The background to this question is that my team is on a shared >>>>>>> Flink >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to >>>>>>> > re-write >>>>>>> > >> all our code as a native Flink job (rather than a remote >>>>>>> stateful >>>>>>> > function) >>>>>>> > >> then I need to get started right away. >>>>>>> > >> >>>>>>> > >> Many thanks, >>>>>>> > >> Fil >>>>>>> > >> >>>>>>> > >>>>>>> > -------------------------- >>>>>>> > Ken Krugler >>>>>>> > http://www.scaleunlimited.com >>>>>>> > Custom big data solutions >>>>>>> > Flink, Pinot, Solr, Elasticsearch >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> >>>>>>
