[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750371#comment-17750371 ] Omnia Ibrahim edited comment on KAFKA-15102 at 8/2/23 3:06 PM: --- The KIP got accepted and the PR is ready was (Author: omnia_h_ibrahim): The KIP is accepted now and the PR is ready > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Assignee: Omnia Ibrahim >Priority: Major > Fix For: 3.3, 3.2.4, 3.1.3, 3.6.0, 3.4.2, 3.5.2 > > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750371#comment-17750371 ] Omnia Ibrahim edited comment on KAFKA-15102 at 8/2/23 3:06 PM: --- The KIP is accepted now and the PR is ready was (Author: omnia_h_ibrahim): The KIP is accepted and the PR is ready > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Assignee: Omnia Ibrahim >Priority: Major > Fix For: 3.3, 3.2.4, 3.1.3, 3.6.0, 3.4.2, 3.5.2 > > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739924#comment-17739924 ] Omnia Ibrahim edited comment on KAFKA-15102 at 7/4/23 1:29 PM: --- [~ChrisEgerton] I updated the compatibility section in the KIP with the impacted versions and linked to this JIRA. I can open a small KIP to have `{{{}replication.policy.internal.topic.separator.enabled` if you don't have time to do it. {}}} was (Author: omnia_h_ibrahim): [~ChrisEgerton] I updated the compatibility section in the KIP with the impacted versions and linked to this JIRA. I can open a small KIP to have `{{{}replication.policy.internal.topic.separator.enabled` if you don't have time to do it. {}}} > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738744#comment-17738744 ] Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:30 PM: Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option.{}}} {{{}We can also update the backward compatibility section in KIP-690 to ask users to provide a custom implementation that the override `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. was (Author: omnia_h_ibrahim): Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option.{}}} {{{}We can also update the backward compatibility section in KIP-690 to ask users to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738744#comment-17738744 ] Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:01 PM: Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option.{}}} {{{}We can also update the backward compatibility section in KIP-690 to ask users to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. was (Author: omnia_h_ibrahim): Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option.{}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738744#comment-17738744 ] Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM: Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. was (Author: omnia_h_ibrahim): Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}}{{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738744#comment-17738744 ] Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM: Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option.{}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. was (Author: omnia_h_ibrahim): Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738744#comment-17738744 ] Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM: Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. was (Author: omnia_h_ibrahim): Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738744#comment-17738744 ] Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM: Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. was (Author: omnia_h_ibrahim): Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in the KIP accounted only for using the default separator configuration and didn't address the usage custom separator (my mistake here). [~ChrisEgerton] I think having `{{{}replication.policy.internal.topic.separator.enabled` is a good option. {}}} {{{}We can also update the backward compatibility section in KIP-690 to ask customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and `ReplicationPolicy.checkpointsTopic` methods to use old topics if they still want to use the old internal topics or delete them so they are aware of this issue. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738733#comment-17738733 ] Chris Egerton edited comment on KAFKA-15102 at 6/29/23 6:43 PM: [~ddufour1a] Thanks for raising this. I believe if we wanted to preserve backward compatibility perfectly, we would have had to ignore the custom separator when creating topics affected by KIP-690, and possibly introduced separate opt-in configuration logic to disable that behavior (i.e., resume taking custom separators into account, even for topics that were not affected prior to KIP-690). However, since 3.1.0 came out a year and a half ago, things are less cut-and-dry: there may be more negative fallout for users who are accustomed to the current behavior than what's currently being experienced by those who are used to the previous behavior. One possibility could be to revert to older behavior for the remainder of our 3.x.y releases with a single configuration property such as {{replication.policy.internal.topic.separator.enabled}} that defaults to {{false}} for now and, come 4.0, defaults to {{{}true{}}}. [~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to approach this accidental break in compatibility? It's probably worth bringing this up in a discussion thread on the user/dev mailing lists, but I'd rather get some feedback on a potential fix before bringing this before a larger audience. was (Author: chrisegerton): [~ddufour1a] Thanks for raising this. I believe if we wanted to preserve backward compatibility perfectly, we would have had to ignore the custom separator when creating topics affected by KIP-690, and possibly introduced separate opt-in configuration logic to disable that behavior (i.e., resume taking custom separators into account, even for topics that were not affected prior to KIP-690). However, since 3.1.0 came out a year and a half ago, things are less cut-and-dry: there may be more negative fallout for users who are accustomed to the current behavior than what's currently being experienced by those who are used to the previous behavior. One possibility could be to revert to older behavior for the remainder of our 3.x.y releases with a single configuration property such as {{replication.policy.internal.topic.separator.enabled}} that defaults to {{false}} for now and, come 4.0, defaults to {{{}true{}}}. [~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to approach this accidental break in compatibility? > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)