Hello, I have two streams A and B. I need to enrich events coming from stream B with events coming from A and I store events coming from A in a key-value store to enrich events from B. Events that doesn't get enriched are sent to a deferred queue(kafka stream) and are read back later.
Most of the the time the events from Stream B are sent to defer queue because of bit delay in storing the events into a key-value store from Stream A and events coming into A and B are almost real time. I want to introduce a delay into reading into my spout reading from Stream B so as to make sure higher % of events get enriched in first shot rather than getting enriched post reading from defer queue. I tried putting a check on the lag and controlling on the backlog queue to get a hold but didn't seemed right and would enter into draining and other issues. Is there a way in the kafka consumer or Storm spout to control the data in flow to come with delay for processing? Thanks, -sandeep.
