Xu Mingmin created STORM-2340: --------------------------------- Summary: KafkaSpout is blocked in AutoCommitMode Key: STORM-2340 URL: https://issues.apache.org/jira/browse/STORM-2340 Project: Apache Storm Issue Type: Improvement Components: storm-kafka-client Reporter: Xu Mingmin
What's the issue? When Storm topology is run on 'At-Most-Once' mode, with ack executor=0, KafkaSpout cannot emit records after some time. What's the cause? Function poll() controls whether it need to pull more data from Kafka cluster. With the condition !waitingToEmit() && numUncommittedOffsets < maxUncommittedOffsets, it's always FALSE after reaching threshold, as numUncommittedOffsets is creasing and never reset on 'At-Most-Once' mode. What's the solution? I change the condition to !waitingToEmit() && (numUncommittedOffsets < maxUncommittedOffsets || consumerAutoCommitMode), that return TRUE with AutoCommitMode regarding of UnCommittedOffset, see the comments inline. Also, it's not required to track emitted(msgId) and numUncommittedOffsets with AutoCommitMode . -- This message was sent by Atlassian JIRA (v6.3.15#6346)