Greg Harris created KAFKA-14663: ----------------------------------- Summary: High throughput topics can starve low-throughput MM2 offset syncs Key: KAFKA-14663 URL: https://issues.apache.org/jira/browse/KAFKA-14663 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 3.3.0, 3.0.0, 3.1.0, 3.4.0, 3.5.0 Reporter: Greg Harris Assignee: Greg Harris
In MM2, a semaphore is used to throttle the number of offset syncs written to the offset-syncs topic. If too many offset writes are requested (for example, from high-throughput topics) then some are silently dropped and never retried. This is acceptable for a single topic-partition, where a later record may re-trigger the offset-sync and write the sync successfully. However, if there is a large variance between throughput in the topics emitted by an MM2 instance, it is possible for high-throughput topics to trigger many offset syncs, and cause the offset-syncs for a co-located low-throughput topic to be unfairly dropped. This can cause the downstream offsets for the starved topic to lag behind significantly, or be prevented completely. Instead, we should have some sort of fairness mechanism where low-thoughput topics are given similar priority to high-throughput topics in producing offset syncs, and cause excess sync messages from high-throughput topics to be dropped instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)