[ https://issues.apache.org/jira/browse/STORM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070819#comment-16070819 ]
Hugo Louro commented on STORM-2430: ----------------------------------- [~Srdo] you have done several improvements related to this. Do you think this bug is fixed? > Potential Race condition in Kafka Spout > --------------------------------------- > > Key: STORM-2430 > URL: https://issues.apache.org/jira/browse/STORM-2430 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client > Affects Versions: 1.0.2, 1.x > Reporter: Kushal Bhatt > > Kafka spout hangs when the number of uncommitted messages exceeds the max > allowed uncommitted messages and some intermediate tuples have failed in down > stream bolt. > Steps of reproduction. > Create a simple topology with one kafka spout and a slow bolt. > In kafka spout set the maximum uncommitted messages to a small number like > 100. > Bolt should process 10 tuples in second. And program it to fail on some > random tuples. For eg: say tuple number 10 fails. Also assume there is only > 1 Kafka partition the spout reads from. > Spout on first execution of nextTuple() gets 110 records and emits them. At > this point number of uncommitted message would be 110. > First 9 tuples are acked by the bolt. 10th tuple is failed by the bolt. > KafkaSpout puts it on retry queue. > Tuple number 11 to 110 are acked by bolt . But spout only commits till offset > 9.[link | > https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L510] > Now, the number of uncommitted messages = 110 - 9 = 101 > 100 (max allowed > uncommitted messages) > No new records are polled from kafka.[link | > https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L239]. > The spout is stuck as the nothing is polled. > Solution is to explicitly go through retry queue explicitly and emit tuples > that are ready on every nextTuple(). -- This message was sent by Atlassian JIRA (v6.4.14#64029)