[ https://issues.apache.org/jira/browse/STORM-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400333#comment-16400333 ]
RAbreu commented on STORM-2994: ------------------------------- I've been injecting in the Kafka topic two types of messages * messages that are incorrect and won't be emitted by the Spout * messages that are correct and will be emitted by the Spout Using storm-kafka-clients 1.1.0, even the messages that are emitted pile up and aren't commited {code:java} 2018-03-15T12:22:25.880Z o.a.s.k.s.KafkaSpout [DEBUG] - Polled [0] records from Kafka. [1379] uncommitted offsets across all topic partitions {code} The incorrect messages aren't added to the total of uncommitted messages. Using storm-kafka-clients 1.1.2, the behaviour is different. I uploaded the same topology, bumping storm-kafka-clients to 1.2.1, picking up the previous lag of > 14000 and reducing it to ~3000. It didn't go to 0 and stayed firmly in that level. Once I started injecting correct messages, eventually all messages are committed (not sure at this point if it is when he commit interval elapses) Question: Aren't the tuples to which > KafkaSpout consumes messages but doesn't commit offsets > ------------------------------------------------------- > > Key: STORM-2994 > URL: https://issues.apache.org/jira/browse/STORM-2994 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client > Affects Versions: 1.1.0, 1.1.2 > Reporter: RAbreu > Priority: Major > > A topology that consumes from two different Kafka clusters: 0.10.1.1 and > 0.10.2.1. > Spouts consuming from 0.10.2.1 have a low lag (and regularly commit offsets) > The Spout that consumes from 0.10.1.1 exhibits either: > 1- Unknown lag > 2- Lag that increments as the Spout reads messages from Kafka > > In DEBUG, Offset manager logs: "topic-partition has NO offsets ready to be > committed", despite continuing to consume messages. > Several configuration tweaks were tried, including setting maxRetries to 1, > in case messages with a lower offset were being retried (logs didn't show it, > though) > offsetCommitPeriodMs was also lowered to no avail. > The only configuration that works is to have > ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG=true, but this is undesired since > we lose processing guarantees. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)