Hello storm users, We have a simple topology with one kafka-spout and a series of 5 bolts.
First bolt receives ~3K tuples/second from kafka-spout and emits only ~500/second. The last bolt processes very less tuples (~100 a hour) and it sends output to kafka. The execute latencies in all bolts are very low. The issue we are seeing: After the last bolt processes about 2500 tuples, it stops receiving anything. So all those tuples which are emitted by the previous bolt do not get acked and it shows up as failed messages in kafka-spout. Now kafka-spout starts replaying these failed tuples and it keeps failing every time because the last bolt is not processing anything at all. If we restart the topology, it starts processing normally till the last bolt again hits around 2500 tuples. This has happened 2-3 times and every time the number of messages after it stop is around 2500. In all the bolts we are acking all tuples (catching throwable everywhere and logging it and acking the tuple). We don’t see any failures in any of the bolts. Has anyone ever seen anything similar? What could be causing this last bolt to not receive any tuples at all after a certain number of messages? Thanks, Soumi
