Hello, I am thinking of using the Kafka Steams feature to "unify" our real-time and scheduled workflow. An example is that in our workflow with stages A--> B --> C, the A --> B segment can be achieved in real-time, but B-->C segment is usually a done with a scheduled job, running maybe once per hour or once per 5 minutes, etc.
I am hoping to model this using Kafka Streams. Each stage would be a topic: the Kafka Streams will process real-time events in topic-A and send result to topic-B. The challenge is when I process the events in topic-B, I want to be able to process each event with a crontab-like schedule, so that if the process is successful (by checking an external API) the event is send to topic-C, otherwise, we will re-process the event again according to the schedule. Can I use the RocksDB key/value state store to store the topic-B events that failed to process, and have a scheduler (like quartz scheduler) to iterate all events in the store and re-process again? I know I can always keep the state outside of Kafka but I like that the state store is fault-tolerant and can be rebuilt automatically if the instance fails. The examples I found so far seems to imply that the state store is only accessible from within a processor. Thanks, Yi
