Siddharth Anand created KAFKA-14757:
---------------------------------------
Summary: Kafka Cooperative Sticky Assignor results in significant
duplicate consumption
Key: KAFKA-14757
URL: https://issues.apache.org/jira/browse/KAFKA-14757
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 3.1.1
Environment: AWS MSK (broker) and Spring Kafka (2.8.7) for use in
Spring Boot consumers.
Reporter: Siddharth Anand
Details may be found within the linked document:
[Kafka Cooperative Sticky Assignor Issue : Duplicate Consumption |
[https://docs.google.com/document/d/1E7qAwGOpF8jo_YhF4NwUx9CXxUGJmT8OhHEqIg7-GfI/edit?usp=sharing]]
In a nutshell, we noticed that the Cooperative Sticky Assignor resulted in
significant duplicate message consumption. During last year's F1 Grand Prix
events and World Cup soccer events, our company's Kafka-based platform received
live-traffic. This live traffic, coupled with autoscaled consumers resulted in
as much as 70% duplicate message consumption at the Kafka consumers.
In December 2022, we ran a synthetic load test to confirm that duplicate
message consumption occurs during consumer scale out/in and Kafka partition
rebalancing when using the Cooperative Sticky Assignor. This issue does not
occur when using the Range Assignor.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
