multi tenant workflow execution
Hi there,
I am researching running one flink job to support customized event driven
workflow executions. The use case is to support running various workflows
that listen to a set of kafka topics and performing various rpc checks, a
user travel through multiple stages in a rule execution(workflow
execution). e.g
kafka topic : user click stream
rpc checks:
if user is member,
if user has shown interest of signup
workflows:
workflow 1: user click -> if user is member do A then do B
workflow 2: user click -> if user has shown interest of signup then do A
otherwise wait for 60 mins and try recheck, expire in 24 hours
The goal is as I said to run workflow1 & workflow2 in one flink job.
Initial thinking describes below
sources are series of kafka topics, all events go through coMap,cache
lookup event -> rules mapping and fan out to multiple {rule, user} tuple.
Based on rule definition and stage user is in a given rule, it do series of
async rpc check and side outputs to various of sinks.
- If a {rule, user} tuple needs to stay in a operator states longer (1
day), there should be a window following async rpc checks with customized
purgetrigger firing those passes and purge either pass check or expired
tuples.
- If a {rule, user} execute to a stage which waits for a kafka event, it
should be added to cache and hookup with coMap lookups near sources
Does that makes sense?
Thanks,
Chen
Re: multi tenant workflow execution
Hi Chen,
if you plan to implement your application on top of the upcoming Flink
1.2.0 release, you might find the new AsyncFunction [1] and the
ProcessFunction [2] helpful.
AsyncFunction can be used for non-blocking calls to external services and
maintains the checkpointing semantics.
ProcessFunction allows to register and react to timers. This might easier
to use than a window for the 24h timeout.
Best,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/asyncio.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html
2017-01-24 0:41 GMT+01:00 Chen Qin :
> Hi there,
>
> I am researching running one flink job to support customized event driven
> workflow executions. The use case is to support running various workflows
> that listen to a set of kafka topics and performing various rpc checks, a
> user travel through multiple stages in a rule execution(workflow
> execution). e.g
>
> kafka topic : user click stream
> rpc checks:
>
> if user is member,
> if user has shown interest of signup
>
>
> workflows:
>
>
> workflow 1: user click -> if user is member do A then do B
> workflow 2: user click -> if user has shown interest of signup then do A
> otherwise wait for 60 mins and try recheck, expire in 24 hours
>
> The goal is as I said to run workflow1 & workflow2 in one flink job.
>
> Initial thinking describes below
>
> sources are series of kafka topics, all events go through coMap,cache
> lookup event -> rules mapping and fan out to multiple {rule, user} tuple.
> Based on rule definition and stage user is in a given rule, it do series of
> async rpc check and side outputs to various of sinks.
>
>- If a {rule, user} tuple needs to stay in a operator states longer (1
>day), there should be a window following async rpc checks with customized
>purgetrigger firing those passes and purge either pass check or expired
>tuples.
>- If a {rule, user} execute to a stage which waits for a kafka event,
>it should be added to cache and hookup with coMap lookups near sources
>
>
> Does that makes sense?
>
> Thanks,
> Chen
>
>
>
Re: multi tenant workflow execution
Hi Fabian,
AsyncFunction and ProcessFunction do help!
I assume per event timers I created in implement RichProcessFunction will
be part of key grouped states & cached in memory during runtime right? I am
interested in this because we are targeting large deployment of million TPS
event source. I would like to understand checkpoint size and speed
implications.
How about checkpointing iteration stream? Can we achieve at least once
semantic in 1.2 on integration jobs?
Thanks,
Chen
On Tue, Jan 24, 2017 at 2:26 AM, Fabian Hueske wrote:
> Hi Chen,
>
> if you plan to implement your application on top of the upcoming Flink
> 1.2.0 release, you might find the new AsyncFunction [1] and the
> ProcessFunction [2] helpful.
> AsyncFunction can be used for non-blocking calls to external services and
> maintains the checkpointing semantics.
> ProcessFunction allows to register and react to timers. This might easier
> to use than a window for the 24h timeout.
>
> Best,
> Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/dev/stream/asyncio.html
> [2] https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/dev/stream/process_function.html
>
> 2017-01-24 0:41 GMT+01:00 Chen Qin :
>
>> Hi there,
>>
>> I am researching running one flink job to support customized event driven
>> workflow executions. The use case is to support running various workflows
>> that listen to a set of kafka topics and performing various rpc checks, a
>> user travel through multiple stages in a rule execution(workflow
>> execution). e.g
>>
>> kafka topic : user click stream
>> rpc checks:
>>
>> if user is member,
>> if user has shown interest of signup
>>
>>
>> workflows:
>>
>>
>> workflow 1: user click -> if user is member do A then do B
>> workflow 2: user click -> if user has shown interest of signup then do A
>> otherwise wait for 60 mins and try recheck, expire in 24 hours
>>
>> The goal is as I said to run workflow1 & workflow2 in one flink job.
>>
>> Initial thinking describes below
>>
>> sources are series of kafka topics, all events go through coMap,cache
>> lookup event -> rules mapping and fan out to multiple {rule, user} tuple.
>> Based on rule definition and stage user is in a given rule, it do series of
>> async rpc check and side outputs to various of sinks.
>>
>>- If a {rule, user} tuple needs to stay in a operator states longer
>>(1 day), there should be a window following async rpc checks with
>>customized purgetrigger firing those passes and purge either pass check or
>>expired tuples.
>>- If a {rule, user} execute to a stage which waits for a kafka event,
>>it should be added to cache and hookup with coMap lookups near sources
>>
>>
>> Does that makes sense?
>>
>> Thanks,
>> Chen
>>
>>
>>
>
Re: multi tenant workflow execution
Hi Chen,
yes, timers of a ProcessFunction are organized by key (you can have
multiple timers per key as well), stored in the keyed state, checkpointed,
and restored.
I'm not sure about the guarantees for iterative streams.
Best, Fabian
2017-01-25 8:18 GMT+01:00 Chen Qin :
> Hi Fabian,
>
> AsyncFunction and ProcessFunction do help!
>
> I assume per event timers I created in implement RichProcessFunction will
> be part of key grouped states & cached in memory during runtime right? I am
> interested in this because we are targeting large deployment of million TPS
> event source. I would like to understand checkpoint size and speed
> implications.
>
> How about checkpointing iteration stream? Can we achieve at least once
> semantic in 1.2 on integration jobs?
>
> Thanks,
> Chen
>
> On Tue, Jan 24, 2017 at 2:26 AM, Fabian Hueske wrote:
>
>> Hi Chen,
>>
>> if you plan to implement your application on top of the upcoming Flink
>> 1.2.0 release, you might find the new AsyncFunction [1] and the
>> ProcessFunction [2] helpful.
>> AsyncFunction can be used for non-blocking calls to external services and
>> maintains the checkpointing semantics.
>> ProcessFunction allows to register and react to timers. This might easier
>> to use than a window for the 24h timeout.
>>
>> Best,
>> Fabian
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/
>> dev/stream/asyncio.html
>> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/
>> dev/stream/process_function.html
>>
>> 2017-01-24 0:41 GMT+01:00 Chen Qin :
>>
>>> Hi there,
>>>
>>> I am researching running one flink job to support customized event
>>> driven workflow executions. The use case is to support running various
>>> workflows that listen to a set of kafka topics and performing various rpc
>>> checks, a user travel through multiple stages in a rule execution(workflow
>>> execution). e.g
>>>
>>> kafka topic : user click stream
>>> rpc checks:
>>>
>>> if user is member,
>>> if user has shown interest of signup
>>>
>>>
>>> workflows:
>>>
>>>
>>> workflow 1: user click -> if user is member do A then do B
>>> workflow 2: user click -> if user has shown interest of signup then do A
>>> otherwise wait for 60 mins and try recheck, expire in 24 hours
>>>
>>> The goal is as I said to run workflow1 & workflow2 in one flink job.
>>>
>>> Initial thinking describes below
>>>
>>> sources are series of kafka topics, all events go through coMap,cache
>>> lookup event -> rules mapping and fan out to multiple {rule, user} tuple.
>>> Based on rule definition and stage user is in a given rule, it do series of
>>> async rpc check and side outputs to various of sinks.
>>>
>>>- If a {rule, user} tuple needs to stay in a operator states longer
>>>(1 day), there should be a window following async rpc checks with
>>>customized purgetrigger firing those passes and purge either pass check
>>> or
>>>expired tuples.
>>>- If a {rule, user} execute to a stage which waits for a kafka
>>>event, it should be added to cache and hookup with coMap lookups near
>>>sources
>>>
>>>
>>> Does that makes sense?
>>>
>>> Thanks,
>>> Chen
>>>
>>>
>>>
>>
>
