[ 
https://issues.apache.org/jira/browse/KAFKA-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856168#comment-17856168
 ] 

Josep Prat commented on KAFKA-16254:
------------------------------------

Changing target fix version to 3.9 since this is not a blocker and we are past 
code freeze

> Allow MM2 to fully disable offset sync feature
> ----------------------------------------------
>
>                 Key: KAFKA-16254
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16254
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 3.5.0, 3.6.0, 3.7.0
>            Reporter: Omnia Ibrahim
>            Assignee: Omnia Ibrahim
>            Priority: Major
>              Labels: need-kip
>             Fix For: 3.8.0
>
>
> *Background:* 
> At the moment syncing offsets feature in MM2 is broken to 2 parts
>  # One is in `MirrorSourceTask` where we store the new recored's offset on 
> target cluster to {{offset_syncs}} internal topic after mirroring the record. 
> Before KAFKA-14610 in 3.5 MM2 used to just queue the offsets and publish them 
> later but since 3.5 this behaviour changed we now publish any offset syncs 
> that we've queued up, but have not yet been able to publish when 
> `MirrorSourceTask.commit` get invoked. This introduced an over head to commit 
> process.
>  # The second part is in checkpoints source task where we use the new record 
> offsets from {{offset_syncs}} and update {{checkpoints}} and 
> {{__consumer_offsets}} topics.
> *Problem:*
> For customers who only use MM2 for mirroring data and not interested in 
> syncing offsets feature they now can disable the second part of this feature 
> which is by disabling {{emit.checkpoints.enabled}} and/or 
> {{sync.group.offsets.enabled}} to disable emitting {{__consumer_offsets}} 
> topic but nothing disabling 1st part of the feature. 
> The problem get worse if they disabled MM2 from creating offset syncs 
> internal topic as 
> 1. this will increase throughput as MM2 will try to force trying to update 
> the offset with every mirrored batch which impacting the performance of our 
> MM2.
> 2. Get too many error logs because they don't create the sync offset topic as 
> they don't use the feature.
> *Possible solution:*
> Allow customers to fully disable the feature if they don't really need it 
> similar to how we fully can disable other MM2 features like heartbeat feature 
> by adding a new config.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to