Greg Harris created KAFKA-14663:
-----------------------------------
Summary: High throughput topics can starve low-throughput MM2
offset syncs
Key: KAFKA-14663
URL: https://issues.apache.org/jira/browse/KAFKA-14663
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 3.3.0, 3.0.0, 3.1.0, 3.4.0, 3.5.0
Reporter: Greg Harris
Assignee: Greg Harris
In MM2, a semaphore is used to throttle the number of offset syncs written to
the offset-syncs topic. If too many offset writes are requested (for example,
from high-throughput topics) then some are silently dropped and never retried.
This is acceptable for a single topic-partition, where a later record may
re-trigger the offset-sync and write the sync successfully.
However, if there is a large variance between throughput in the topics emitted
by an MM2 instance, it is possible for high-throughput topics to trigger many
offset syncs, and cause the offset-syncs for a co-located low-throughput topic
to be unfairly dropped.
This can cause the downstream offsets for the starved topic to lag behind
significantly, or be prevented completely.
Instead, we should have some sort of fairness mechanism where low-thoughput
topics are given similar priority to high-throughput topics in producing offset
syncs, and cause excess sync messages from high-throughput topics to be dropped
instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
