Ruslan Gryn created KAFKA-10711: ----------------------------------- Summary: A low value in commit.interval.ms leads to unnecessary committing offsets Key: KAFKA-10711 URL: https://issues.apache.org/jira/browse/KAFKA-10711 Project: Kafka Issue Type: Improvement Components: consumer, offset manager Affects Versions: 2.6.0 Reporter: Ruslan Gryn
We want to avoid double delivery of the same records in Kafka. Therefore, we decided to set {code:java} commit.interval.ms=0 and max.poll.records=1{code} . Because default commit.interval.ms= 5 sec. That's why, if the app crashed in runtime then after restarting the app will receive all uncommitted records for 5 sec. But when committing every record then the app will receive only 1 duplicated record. We are expecting that the consumer will poll(5 sec) a single record from the topic and after the next poll(5 sec), the consumer will commit offset of the record from the previous poll. But the consumer commits offsets without any delays even if offsets were already committed before. That's why such a high volume of commits overload Kafka Brokers. Could you please improve the behavior of consumers to avoid committing offsets that were already committed before and only commit offset if necessary? -- This message was sent by Atlassian Jira (v8.3.4#803005)