Sure thing. One question -- Flink 1.16 was just released a few days ago. Should I support 1.15, or just go straight to 1.16?
On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrm...@apache.org> wrote: > Hi folks, > > if you can open a PR for supporting Flink 1.15 Galen, then this would be > awesome. I've assigned you to this ticket. The next thing after merging > this PR would be creating a new StateFun release. Once we have merged the > PR, let's check who can help with it the fastest. > > Cheers, > Till > > On Mon, Oct 31, 2022 at 1:10 PM Galen Warren <ga...@cvillewarrens.com> > wrote: > >> Yes, I could do that. >> >> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <filip.karni...@gmail.com> >> wrote: >> >>> Hi All >>> >>> So what's the play here? >>> >>> Galen, what do you think about taking this on? Perhaps ++Till would >>> assign this jira to you (with your permission) given he's helped me out >>> with statefun work before >>> https://issues.apache.org/jira/browse/FLINK-29814 >>> >>> I can try to move to move statefun to flink 1.16 when it's out >>> >>> >>> Kind regards >>> Fil >>> >>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <filip.karni...@gmail.com> >>> wrote: >>> >>>> Hi All >>>> >>>> Our use case is that we need to process elements for the same key >>>> sequentially, and this processing involves async operations. >>>> >>>> If any part of the processing fails, we store the offending and all >>>> subsequent incoming messages for that key in the state and not process any >>>> further messages for that key, until a retry succeeds or a human sends a >>>> 'skip' command message. >>>> >>>> diagram: >>>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >>>> mermaid (in case mermaid.live goes down in the future): >>>> graph LR >>>> incoming[incoming events] --> job(Flink statefun job) >>>> commands[commands] -->|skip| job >>>> job --> |sequentially per key| remote(remote function) >>>> remote --> |on failure, delayed message to retry| remote >>>> remote --> |async puts/gets with side effects| other(other systems) >>>> >>>> Having the processing happen outside of Flink is nice-to-have from an >>>> independent scalability point of view, but is not strictly required. >>>> >>>> So long story short - no cyclic messaging, but also no way I can think >>>> of to use existing native Flink operators like async i/o (which when I last >>>> checked a few years back didn't have access to keyed state) >>>> >>>> >>>> P.S. Please note that there is already a pull request that has >>>> something to do wtih Flink 1.15, albeit without a description or a jira: >>>> https://github.com/apache/flink-statefun/pull/314 >>>> >>>> >>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com> >>>> wrote: >>>> >>>>> Hi Gordon (and others), >>>>> >>>>> I'm also using this project for stateful messaging, including messaging >>>>> among functions. >>>>> >>>>> I've contributed a small amount of code in the past and have also >>>>> enabled >>>>> Flink 1.15 compatibility in a local fork, so I might be able to help >>>>> out >>>>> here. >>>>> >>>>> Thanks, >>>>> Galen >>>>> >>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < >>>>> kkrugler_li...@transpac.com> >>>>> wrote: >>>>> >>>>> > Hi Gordon, >>>>> > >>>>> > We’re using it for stateful messaging, and also calling remote >>>>> > Python-based functions. >>>>> > >>>>> > So yes, also very interested in what is going to happen with the this >>>>> > subproject in the future. >>>>> > >>>>> > — Ken >>>>> > >>>>> > >>>>> > >>>>> > > Begin forwarded message: >>>>> > > >>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT >>>>> > > To: dev@flink.apache.org >>>>> > > Reply-To: dev@flink.apache.org >>>>> > > >>>>> > > Hi Filip, >>>>> > > >>>>> > > Thanks for bringing this up. >>>>> > > >>>>> > > The hard truth is that committers who were previously active on the >>>>> > > StateFun subproject, including myself, all currently have other >>>>> focuses. >>>>> > > Indeed, we may need to discuss with the community on how to >>>>> proceed if >>>>> > > there seems to be no continued committer coverage. >>>>> > > >>>>> > > If it's just a matter of upgrading the supported Flink version, >>>>> I'm still >>>>> > > familiar enough with the subproject to probably be able to drive >>>>> this (or >>>>> > > if your team is up to it, I can assist you on that). >>>>> > > >>>>> > > For the long-term, as a data point I'm curious to see how many >>>>> users are >>>>> > > using StateFun in production today, and how you're using it? >>>>> > > >>>>> > > - Do your applications have arbitrary / cyclic / bi-directional >>>>> > > messaging between individual functions? >>>>> > > - Or are you utilizing StateFun simply to allow your stateful >>>>> functions >>>>> > > to run remotely as separate processes? >>>>> > > >>>>> > > If the majority is only the latter category, there might be a case >>>>> to >>>>> > > support remote functions natively in Flink (which has been a >>>>> discussion >>>>> > in >>>>> > > the past). >>>>> > > >>>>> > > Thanks, >>>>> > > Gordon >>>>> > > >>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >>>>> filip.karni...@gmail.com >>>>> > > >>>>> > > wrote: >>>>> > > >>>>> > >> Hi, I noticed that the development on stateful functions has come >>>>> to a >>>>> > bit >>>>> > >> of a halt, with a pull request to update statefun to use Flink >>>>> 1.15 >>>>> > being >>>>> > >> in the `open` state since May 2022. >>>>> > >> >>>>> > >> What do we think is the future of this sub-project? >>>>> > >> >>>>> > >> The background to this question is that my team is on a shared >>>>> Flink >>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to >>>>> > re-write >>>>> > >> all our code as a native Flink job (rather than a remote stateful >>>>> > function) >>>>> > >> then I need to get started right away. >>>>> > >> >>>>> > >> Many thanks, >>>>> > >> Fil >>>>> > >> >>>>> > >>>>> > -------------------------- >>>>> > Ken Krugler >>>>> > http://www.scaleunlimited.com >>>>> > Custom big data solutions >>>>> > Flink, Pinot, Solr, Elasticsearch >>>>> > >>>>> > >>>>> > >>>>> > >>>>> >>>>
