I think there might still be value in supporting 1.15 since not everyone upgrades Flink very fast. Hopefully, for Statefun the diff between Flink 1.15 and 1.16 boils down to changing the Flink dependencies.
Cheers, Till On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com> wrote: > Sure thing. One question -- Flink 1.16 was just released a few days ago. > Should I support 1.15, or just go straight to 1.16? > > On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi folks, >> >> if you can open a PR for supporting Flink 1.15 Galen, then this would be >> awesome. I've assigned you to this ticket. The next thing after merging >> this PR would be creating a new StateFun release. Once we have merged the >> PR, let's check who can help with it the fastest. >> >> Cheers, >> Till >> >> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com> >> wrote: >> >>> Yes, I could do that. >>> >>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <filip.karni...@gmail.com> >>> wrote: >>> >>>> Hi All >>>> >>>> So what's the play here? >>>> >>>> Galen, what do you think about taking this on? Perhaps ++Till would >>>> assign this jira to you (with your permission) given he's helped me out >>>> with statefun work before >>>> https://issues.apache.org/jira/browse/FLINK-29814 >>>> >>>> I can try to move to move statefun to flink 1.16 when it's out >>>> >>>> >>>> Kind regards >>>> Fil >>>> >>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <filip.karni...@gmail.com> >>>> wrote: >>>> >>>>> Hi All >>>>> >>>>> Our use case is that we need to process elements for the same key >>>>> sequentially, and this processing involves async operations. >>>>> >>>>> If any part of the processing fails, we store the offending and all >>>>> subsequent incoming messages for that key in the state and not process any >>>>> further messages for that key, until a retry succeeds or a human sends a >>>>> 'skip' command message. >>>>> >>>>> diagram: >>>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >>>>> mermaid (in case mermaid.live goes down in the future): >>>>> graph LR >>>>> incoming[incoming events] --> job(Flink statefun job) >>>>> commands[commands] -->|skip| job >>>>> job --> |sequentially per key| remote(remote function) >>>>> remote --> |on failure, delayed message to retry| remote >>>>> remote --> |async puts/gets with side effects| other(other >>>>> systems) >>>>> >>>>> Having the processing happen outside of Flink is nice-to-have from an >>>>> independent scalability point of view, but is not strictly required. >>>>> >>>>> So long story short - no cyclic messaging, but also no way I can think >>>>> of to use existing native Flink operators like async i/o (which when I >>>>> last >>>>> checked a few years back didn't have access to keyed state) >>>>> >>>>> >>>>> P.S. Please note that there is already a pull request that has >>>>> something to do wtih Flink 1.15, albeit without a description or a jira: >>>>> https://github.com/apache/flink-statefun/pull/314 >>>>> >>>>> >>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com> >>>>> wrote: >>>>> >>>>>> Hi Gordon (and others), >>>>>> >>>>>> I'm also using this project for stateful messaging, including >>>>>> messaging >>>>>> among functions. >>>>>> >>>>>> I've contributed a small amount of code in the past and have also >>>>>> enabled >>>>>> Flink 1.15 compatibility in a local fork, so I might be able to help >>>>>> out >>>>>> here. >>>>>> >>>>>> Thanks, >>>>>> Galen >>>>>> >>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < >>>>>> kkrugler_li...@transpac.com> >>>>>> wrote: >>>>>> >>>>>> > Hi Gordon, >>>>>> > >>>>>> > We’re using it for stateful messaging, and also calling remote >>>>>> > Python-based functions. >>>>>> > >>>>>> > So yes, also very interested in what is going to happen with the >>>>>> this >>>>>> > subproject in the future. >>>>>> > >>>>>> > — Ken >>>>>> > >>>>>> > >>>>>> > >>>>>> > > Begin forwarded message: >>>>>> > > >>>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT >>>>>> > > To: dev@flink.apache.org >>>>>> > > Reply-To: dev@flink.apache.org >>>>>> > > >>>>>> > > Hi Filip, >>>>>> > > >>>>>> > > Thanks for bringing this up. >>>>>> > > >>>>>> > > The hard truth is that committers who were previously active on >>>>>> the >>>>>> > > StateFun subproject, including myself, all currently have other >>>>>> focuses. >>>>>> > > Indeed, we may need to discuss with the community on how to >>>>>> proceed if >>>>>> > > there seems to be no continued committer coverage. >>>>>> > > >>>>>> > > If it's just a matter of upgrading the supported Flink version, >>>>>> I'm still >>>>>> > > familiar enough with the subproject to probably be able to drive >>>>>> this (or >>>>>> > > if your team is up to it, I can assist you on that). >>>>>> > > >>>>>> > > For the long-term, as a data point I'm curious to see how many >>>>>> users are >>>>>> > > using StateFun in production today, and how you're using it? >>>>>> > > >>>>>> > > - Do your applications have arbitrary / cyclic / bi-directional >>>>>> > > messaging between individual functions? >>>>>> > > - Or are you utilizing StateFun simply to allow your stateful >>>>>> functions >>>>>> > > to run remotely as separate processes? >>>>>> > > >>>>>> > > If the majority is only the latter category, there might be a >>>>>> case to >>>>>> > > support remote functions natively in Flink (which has been a >>>>>> discussion >>>>>> > in >>>>>> > > the past). >>>>>> > > >>>>>> > > Thanks, >>>>>> > > Gordon >>>>>> > > >>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >>>>>> filip.karni...@gmail.com >>>>>> > > >>>>>> > > wrote: >>>>>> > > >>>>>> > >> Hi, I noticed that the development on stateful functions has >>>>>> come to a >>>>>> > bit >>>>>> > >> of a halt, with a pull request to update statefun to use Flink >>>>>> 1.15 >>>>>> > being >>>>>> > >> in the `open` state since May 2022. >>>>>> > >> >>>>>> > >> What do we think is the future of this sub-project? >>>>>> > >> >>>>>> > >> The background to this question is that my team is on a shared >>>>>> Flink >>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to >>>>>> > re-write >>>>>> > >> all our code as a native Flink job (rather than a remote stateful >>>>>> > function) >>>>>> > >> then I need to get started right away. >>>>>> > >> >>>>>> > >> Many thanks, >>>>>> > >> Fil >>>>>> > >> >>>>>> > >>>>>> > -------------------------- >>>>>> > Ken Krugler >>>>>> > http://www.scaleunlimited.com >>>>>> > Custom big data solutions >>>>>> > Flink, Pinot, Solr, Elasticsearch >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> >>>>>
