[ https://issues.apache.org/jira/browse/KAFKA-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Egerton resolved KAFKA-14663. ----------------------------------- Resolution: Duplicate > High throughput topics can starve low-throughput MM2 offset syncs > ----------------------------------------------------------------- > > Key: KAFKA-14663 > URL: https://issues.apache.org/jira/browse/KAFKA-14663 > Project: Kafka > Issue Type: Bug > Components: mirrormaker > Affects Versions: 3.1.0, 3.0.0, 3.3.0, 3.4.0, 3.5.0 > Reporter: Greg Harris > Assignee: Greg Harris > Priority: Major > > In MM2, a semaphore is used to throttle the number of offset syncs written to > the offset-syncs topic. If too many offset writes are requested (for example, > from high-throughput topics) then some are silently dropped and never > retried. This is acceptable for a single topic-partition, where a later > record may re-trigger the offset-sync and write the sync successfully. > However, if there is a large variance between throughput in the topics > emitted by an MM2 instance, it is possible for high-throughput topics to > trigger many offset syncs, and cause the offset-syncs for a co-located > low-throughput topic to be unfairly dropped. > This can cause the downstream offsets for the starved topic to lag behind > significantly, or be prevented completely. > Instead, we should have some sort of fairness mechanism where low-thoughput > topics are given similar priority to high-throughput topics in producing > offset syncs, and cause excess sync messages from high-throughput topics to > be dropped instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)