Omnia Ibrahim created KAFKA-16254: ------------------------------------- Summary: Allow MM2 to fully disable offset sync feature Key: KAFKA-16254 URL: https://issues.apache.org/jira/browse/KAFKA-16254 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 3.6.0, 3.5.0, 3.7.0 Reporter: Omnia Ibrahim Assignee: Omnia Ibrahim
*Background:* At the moment syncing offsets feature in MM2 is broken to 2 parts # One is in `MirrorSourceTask` where we store the new recored's offset on target cluster to {{offset_syncs}} internal topic after mirroring the record. Before KAFKA-14610 in 3.5 MM2 used to just queue the offsets and publish them later but since 3.5 this behaviour changed we now publish any offset syncs that we've queued up, but have not yet been able to publish when `MirrorSourceTask.commit` get invoked. This introduced an over head to commit process. # The second part is in checkpoints source task where we use the new record offsets from {{offset_syncs}} and update {{checkpoints}} and {{__consumer_offsets}} topics. *Problem:* For customers who only use MM2 for mirroring data and not interested in syncing offsets feature they now can disable the second part of this feature which is by disabling {{emit.checkpoints.enabled}} and/or {{sync.group.offsets.enabled}} to disable emitting {{__consumer_offsets}} topic but nothing disabling 1st part of the feature. The problem get worse if they disabled MM2 from creating offset syncs internal topic as 1. this will increase throughput as MM2 will try to force trying to update the offset with every mirrored batch which impacting the performance of our MM2. 2. Get too many error logs because they don't create the sync offset topic as they don't use the feature. *Possible solution:* Allow customers to fully disable the feature if they don't really need it similar to how we fully can disable other MM2 features like heartbeat feature by adding a new config. -- This message was sent by Atlassian Jira (v8.20.10#820010)