Yes, I could do that.

On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki <filip.karni...@gmail.com>
wrote:

> Hi All
>
> So what's the play here?
>
> Galen, what do you think about taking this on? Perhaps ++Till would assign
> this jira to you (with your permission) given he's helped me out
> with statefun work before
> https://issues.apache.org/jira/browse/FLINK-29814
>
> I can try to move to move statefun to flink 1.16 when it's out
>
>
> Kind regards
> Fil
>
> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <filip.karni...@gmail.com>
> wrote:
>
>> Hi All
>>
>> Our use case is that we need to process elements for the same key
>> sequentially, and this processing involves async operations.
>>
>> If any part of the processing fails, we store the offending and all
>> subsequent incoming messages for that key in the state and not process any
>> further messages for that key, until a retry succeeds or a human sends a
>> 'skip' command message.
>>
>> diagram:
>> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek
>> mermaid (in case mermaid.live goes down in the future):
>> graph LR
>>     incoming[incoming events] --> job(Flink statefun job)
>>     commands[commands] -->|skip| job
>>     job --> |sequentially per key| remote(remote function)
>>     remote --> |on failure, delayed message to retry| remote
>>     remote --> |async puts/gets with side effects| other(other systems)
>>
>> Having the processing happen outside of Flink is nice-to-have from an
>> independent scalability point of view, but is not strictly required.
>>
>> So long story short - no cyclic messaging, but also no way I can think of
>> to use existing native Flink operators like async i/o (which when I last
>> checked a few years back didn't have access to keyed state)
>>
>>
>> P.S. Please note that there is already a pull request that has something
>> to do wtih Flink 1.15, albeit without a description or a jira:
>> https://github.com/apache/flink-statefun/pull/314
>>
>>
>> On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com>
>> wrote:
>>
>>> Hi Gordon (and others),
>>>
>>> I'm also using this project for stateful messaging, including messaging
>>> among functions.
>>>
>>> I've contributed a small amount of code in the past and have also enabled
>>> Flink 1.15 compatibility in a local fork, so I might be able to help out
>>> here.
>>>
>>> Thanks,
>>> Galen
>>>
>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kkrugler_li...@transpac.com
>>> >
>>> wrote:
>>>
>>> > Hi Gordon,
>>> >
>>> > We’re using it for stateful messaging, and also calling remote
>>> > Python-based functions.
>>> >
>>> > So yes, also very interested in what is going to happen with the this
>>> > subproject in the future.
>>> >
>>> > — Ken
>>> >
>>> >
>>> >
>>> > > Begin forwarded message:
>>> > >
>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards
>>> > > Date: October 26, 2022 at 10:25:26 AM PDT
>>> > > To: dev@flink.apache.org
>>> > > Reply-To: dev@flink.apache.org
>>> > >
>>> > > Hi Filip,
>>> > >
>>> > > Thanks for bringing this up.
>>> > >
>>> > > The hard truth is that committers who were previously active on the
>>> > > StateFun subproject, including myself, all currently have other
>>> focuses.
>>> > > Indeed, we may need to discuss with the community on how to proceed
>>> if
>>> > > there seems to be no continued committer coverage.
>>> > >
>>> > > If it's just a matter of upgrading the supported Flink version, I'm
>>> still
>>> > > familiar enough with the subproject to probably be able to drive
>>> this (or
>>> > > if your team is up to it, I can assist you on that).
>>> > >
>>> > > For the long-term, as a data point I'm curious to see how many users
>>> are
>>> > > using StateFun in production today, and how you're using it?
>>> > >
>>> > >   - Do your applications have arbitrary / cyclic / bi-directional
>>> > >   messaging between individual functions?
>>> > >   - Or are you utilizing StateFun simply to allow your stateful
>>> functions
>>> > >   to run remotely as separate processes?
>>> > >
>>> > > If the majority is only the latter category, there might be a case to
>>> > > support remote functions natively in Flink (which has been a
>>> discussion
>>> > in
>>> > > the past).
>>> > >
>>> > > Thanks,
>>> > > Gordon
>>> > >
>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki <
>>> filip.karni...@gmail.com
>>> > >
>>> > > wrote:
>>> > >
>>> > >> Hi, I noticed that the development on stateful functions has come
>>> to a
>>> > bit
>>> > >> of a halt, with a pull request to update statefun to use Flink 1.15
>>> > being
>>> > >> in the `open` state since May 2022.
>>> > >>
>>> > >> What do we think is the future of this sub-project?
>>> > >>
>>> > >> The background to this question is that my team is on a shared Flink
>>> > >> cluster which will soon be upgrading to Flink 1.15. If I need to
>>> > re-write
>>> > >> all our code as a native Flink job (rather than a remote stateful
>>> > function)
>>> > >> then I need to get started right away.
>>> > >>
>>> > >> Many thanks,
>>> > >> Fil
>>> > >>
>>> >
>>> > --------------------------
>>> > Ken Krugler
>>> > http://www.scaleunlimited.com
>>> > Custom big data solutions
>>> > Flink, Pinot, Solr, Elasticsearch
>>> >
>>> >
>>> >
>>> >
>>>
>>

Reply via email to