This is awesome Eno! Would you mind sharing the JIRA ticket if you have one?
On Sun, Jun 19, 2016 at 12:07 PM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Yi, > > Your observation about accessing the state stores that are already there > vs. keeping state outside of Kafka Streams is a good one. We are currently > working on having the state stores accessible like you mention and should > be able to share some design docs shortly. > > Thanks > Eno > > > On 19 Jun 2016, at 19:49, Yi Chen <y...@symphonycommerce.com> wrote: > > > > Hello, > > > > I am thinking of using the Kafka Steams feature to "unify" our real-time > > and scheduled workflow. An example is that in our workflow with stages > A--> > > B --> C, the A --> B segment can be achieved in real-time, but B-->C > > segment is usually a done with a scheduled job, running maybe once per > hour > > or once per 5 minutes, etc. > > > > I am hoping to model this using Kafka Streams. Each stage would be a > topic: > > the Kafka Streams will process real-time events in topic-A and send > result > > to topic-B. The challenge is when I process the events in topic-B, I want > > to be able to process each event with a crontab-like schedule, so that if > > the process is successful (by checking an external API) the event is send > > to topic-C, otherwise, we will re-process the event again according to > the > > schedule. > > > > Can I use the RocksDB key/value state store to store the topic-B events > > that failed to process, and have a scheduler (like quartz scheduler) to > > iterate all events in the store and re-process again? I know I can always > > keep the state outside of Kafka but I like that the state store is > > fault-tolerant and can be rebuilt automatically if the instance fails. > The > > examples I found so far seems to imply that the state store is only > > accessible from within a processor. > > > > Thanks, > > Yi > >
