[jira] [Updated] (KAFKA-16254) Allow MM2 to fully disable offset sync feature
[
https://issues.apache.org/jira/browse/KAFKA-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josep Prat updated KAFKA-16254:
---
Fix Version/s: 3.9.0
(was: 3.8.0)
> Allow MM2 to fully disable offset sync feature
> --
>
> Key: KAFKA-16254
> URL: https://issues.apache.org/jira/browse/KAFKA-16254
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.5.0, 3.6.0, 3.7.0
>Reporter: Omnia Ibrahim
>Assignee: Omnia Ibrahim
>Priority: Major
> Labels: need-kip
> Fix For: 3.9.0
>
>
> *Background:*
> At the moment syncing offsets feature in MM2 is broken to 2 parts
> # One is in `MirrorSourceTask` where we store the new recored's offset on
> target cluster to {{offset_syncs}} internal topic after mirroring the record.
> Before KAFKA-14610 in 3.5 MM2 used to just queue the offsets and publish them
> later but since 3.5 this behaviour changed we now publish any offset syncs
> that we've queued up, but have not yet been able to publish when
> `MirrorSourceTask.commit` get invoked. This introduced an over head to commit
> process.
> # The second part is in checkpoints source task where we use the new record
> offsets from {{offset_syncs}} and update {{checkpoints}} and
> {{__consumer_offsets}} topics.
> *Problem:*
> For customers who only use MM2 for mirroring data and not interested in
> syncing offsets feature they now can disable the second part of this feature
> which is by disabling {{emit.checkpoints.enabled}} and/or
> {{sync.group.offsets.enabled}} to disable emitting {{__consumer_offsets}}
> topic but nothing disabling 1st part of the feature.
> The problem get worse if they disabled MM2 from creating offset syncs
> internal topic as
> 1. this will increase throughput as MM2 will try to force trying to update
> the offset with every mirrored batch which impacting the performance of our
> MM2.
> 2. Get too many error logs because they don't create the sync offset topic as
> they don't use the feature.
> *Possible solution:*
> Allow customers to fully disable the feature if they don't really need it
> similar to how we fully can disable other MM2 features like heartbeat feature
> by adding a new config.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-16254) Allow MM2 to fully disable offset sync feature
[
https://issues.apache.org/jira/browse/KAFKA-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omnia Ibrahim updated KAFKA-16254:
--
Fix Version/s: 3.8.0
> Allow MM2 to fully disable offset sync feature
> --
>
> Key: KAFKA-16254
> URL: https://issues.apache.org/jira/browse/KAFKA-16254
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.5.0, 3.6.0, 3.7.0
>Reporter: Omnia Ibrahim
>Assignee: Omnia Ibrahim
>Priority: Major
> Labels: need-kip
> Fix For: 3.8.0
>
>
> *Background:*
> At the moment syncing offsets feature in MM2 is broken to 2 parts
> # One is in `MirrorSourceTask` where we store the new recored's offset on
> target cluster to {{offset_syncs}} internal topic after mirroring the record.
> Before KAFKA-14610 in 3.5 MM2 used to just queue the offsets and publish them
> later but since 3.5 this behaviour changed we now publish any offset syncs
> that we've queued up, but have not yet been able to publish when
> `MirrorSourceTask.commit` get invoked. This introduced an over head to commit
> process.
> # The second part is in checkpoints source task where we use the new record
> offsets from {{offset_syncs}} and update {{checkpoints}} and
> {{__consumer_offsets}} topics.
> *Problem:*
> For customers who only use MM2 for mirroring data and not interested in
> syncing offsets feature they now can disable the second part of this feature
> which is by disabling {{emit.checkpoints.enabled}} and/or
> {{sync.group.offsets.enabled}} to disable emitting {{__consumer_offsets}}
> topic but nothing disabling 1st part of the feature.
> The problem get worse if they disabled MM2 from creating offset syncs
> internal topic as
> 1. this will increase throughput as MM2 will try to force trying to update
> the offset with every mirrored batch which impacting the performance of our
> MM2.
> 2. Get too many error logs because they don't create the sync offset topic as
> they don't use the feature.
> *Possible solution:*
> Allow customers to fully disable the feature if they don't really need it
> similar to how we fully can disable other MM2 features like heartbeat feature
> by adding a new config.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
