Hi Chen, if you plan to implement your application on top of the upcoming Flink 1.2.0 release, you might find the new AsyncFunction [1] and the ProcessFunction [2] helpful. AsyncFunction can be used for non-blocking calls to external services and maintains the checkpointing semantics. ProcessFunction allows to register and react to timers. This might easier to use than a window for the 24h timeout.
Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/asyncio.html [2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html 2017-01-24 0:41 GMT+01:00 Chen Qin <qinnc...@gmail.com>: > Hi there, > > I am researching running one flink job to support customized event driven > workflow executions. The use case is to support running various workflows > that listen to a set of kafka topics and performing various rpc checks, a > user travel through multiple stages in a rule execution(workflow > execution). e.g > > kafka topic : user click stream > rpc checks: > > if user is member, > if user has shown interest of signup > > > workflows: > > > workflow 1: user click -> if user is member do A then do B > workflow 2: user click -> if user has shown interest of signup then do A > otherwise wait for 60 mins and try recheck, expire in 24 hours > > The goal is as I said to run workflow1 & workflow2 in one flink job. > > Initial thinking describes below > > sources are series of kafka topics, all events go through coMap,cache > lookup event -> rules mapping and fan out to multiple {rule, user} tuple. > Based on rule definition and stage user is in a given rule, it do series of > async rpc check and side outputs to various of sinks. > > - If a {rule, user} tuple needs to stay in a operator states longer (1 > day), there should be a window following async rpc checks with customized > purgetrigger firing those passes and purge either pass check or expired > tuples. > - If a {rule, user} execute to a stage which waits for a kafka event, > it should be added to cache and hookup with coMap lookups near sources > > > Does that makes sense? > > Thanks, > Chen > > >
