Yes, I could do that. On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <filip.karni...@gmail.com> wrote:
> Hi All > > So what's the play here? > > Galen, what do you think about taking this on? Perhaps ++Till would assign > this jira to you (with your permission) given he's helped me out > with statefun work before > https://issues.apache.org/jira/browse/FLINK-29814 > > I can try to move to move statefun to flink 1.16 when it's out > > > Kind regards > Fil > > On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <filip.karni...@gmail.com> > wrote: > >> Hi All >> >> Our use case is that we need to process elements for the same key >> sequentially, and this processing involves async operations. >> >> If any part of the processing fails, we store the offending and all >> subsequent incoming messages for that key in the state and not process any >> further messages for that key, until a retry succeeds or a human sends a >> 'skip' command message. >> >> diagram: >> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >> mermaid (in case mermaid.live goes down in the future): >> graph LR >> incoming[incoming events] --> job(Flink statefun job) >> commands[commands] -->|skip| job >> job --> |sequentially per key| remote(remote function) >> remote --> |on failure, delayed message to retry| remote >> remote --> |async puts/gets with side effects| other(other systems) >> >> Having the processing happen outside of Flink is nice-to-have from an >> independent scalability point of view, but is not strictly required. >> >> So long story short - no cyclic messaging, but also no way I can think of >> to use existing native Flink operators like async i/o (which when I last >> checked a few years back didn't have access to keyed state) >> >> >> P.S. Please note that there is already a pull request that has something >> to do wtih Flink 1.15, albeit without a description or a jira: >> https://github.com/apache/flink-statefun/pull/314 >> >> >> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com> >> wrote: >> >>> Hi Gordon (and others), >>> >>> I'm also using this project for stateful messaging, including messaging >>> among functions. >>> >>> I've contributed a small amount of code in the past and have also enabled >>> Flink 1.15 compatibility in a local fork, so I might be able to help out >>> here. >>> >>> Thanks, >>> Galen >>> >>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kkrugler_li...@transpac.com >>> > >>> wrote: >>> >>> > Hi Gordon, >>> > >>> > We’re using it for stateful messaging, and also calling remote >>> > Python-based functions. >>> > >>> > So yes, also very interested in what is going to happen with the this >>> > subproject in the future. >>> > >>> > — Ken >>> > >>> > >>> > >>> > > Begin forwarded message: >>> > > >>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >>> > > Date: October 26, 2022 at 10:25:26 AM PDT >>> > > To: dev@flink.apache.org >>> > > Reply-To: dev@flink.apache.org >>> > > >>> > > Hi Filip, >>> > > >>> > > Thanks for bringing this up. >>> > > >>> > > The hard truth is that committers who were previously active on the >>> > > StateFun subproject, including myself, all currently have other >>> focuses. >>> > > Indeed, we may need to discuss with the community on how to proceed >>> if >>> > > there seems to be no continued committer coverage. >>> > > >>> > > If it's just a matter of upgrading the supported Flink version, I'm >>> still >>> > > familiar enough with the subproject to probably be able to drive >>> this (or >>> > > if your team is up to it, I can assist you on that). >>> > > >>> > > For the long-term, as a data point I'm curious to see how many users >>> are >>> > > using StateFun in production today, and how you're using it? >>> > > >>> > > - Do your applications have arbitrary / cyclic / bi-directional >>> > > messaging between individual functions? >>> > > - Or are you utilizing StateFun simply to allow your stateful >>> functions >>> > > to run remotely as separate processes? >>> > > >>> > > If the majority is only the latter category, there might be a case to >>> > > support remote functions natively in Flink (which has been a >>> discussion >>> > in >>> > > the past). >>> > > >>> > > Thanks, >>> > > Gordon >>> > > >>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >>> filip.karni...@gmail.com >>> > > >>> > > wrote: >>> > > >>> > >> Hi, I noticed that the development on stateful functions has come >>> to a >>> > bit >>> > >> of a halt, with a pull request to update statefun to use Flink 1.15 >>> > being >>> > >> in the `open` state since May 2022. >>> > >> >>> > >> What do we think is the future of this sub-project? >>> > >> >>> > >> The background to this question is that my team is on a shared Flink >>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to >>> > re-write >>> > >> all our code as a native Flink job (rather than a remote stateful >>> > function) >>> > >> then I need to get started right away. >>> > >> >>> > >> Many thanks, >>> > >> Fil >>> > >> >>> > >>> > -------------------------- >>> > Ken Krugler >>> > http://www.scaleunlimited.com >>> > Custom big data solutions >>> > Flink, Pinot, Solr, Elasticsearch >>> > >>> > >>> > >>> > >>> >>
