[jira] [Updated] (KAFKA-15465) MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
[
https://issues.apache.org/jira/browse/KAFKA-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mickael Maison updated KAFKA-15465:
---
Fix Version/s: (was: 3.5.2)
(was: 3.6.1)
> MM2 not working when its internal topics are pre-created on a cluster that
> disallows topic creation
> ---
>
> Key: KAFKA-15465
> URL: https://issues.apache.org/jira/browse/KAFKA-15465
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.4.1
>Reporter: Ahmed HIBOT
>Assignee: Omnia Ibrahim
>Priority: Major
> Fix For: 3.7.0
>
>
> h1. Replication steps
> * Setup a source kafka cluster (alias SOURCE) which doesn't allow topic
> creation to MM2 (therefore it doesn't allow the creation of internal topics)
> * Create MM2 internal topics in the source kafka cluster
> * Setup a target kafka cluster (alias TARGET)
> * Enable one way replication SOURCE->TARGET
> MM2 will attempt to create or find its internal topics on the source cluster
> but it will fail with the following stack trace
> {code:java}
> {"log_timestamp": "2023-09-13T09:39:25.612+", "log_level": "ERROR",
> "process_id": 1, "process_name": "mirror-maker", "thread_id": 1,
> "thread_name": "Scheduler for MirrorSourceConnector-creating upstream
> offset-syncs topic", "action_name":
> "org.apache.kafka.connect.mirror.Scheduler", "log_message": "Scheduler for
> MirrorSourceConnector caught exception in scheduled task: creating upstream
> offset-syncs topic"}
> org.apache.kafka.connect.errors.ConnectException: Error while attempting to
> create/find topic 'mm2-offset-syncs.TARGET.internal'
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:155)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:161)
> at
> org.apache.kafka.connect.mirror.MirrorSourceConnector.createOffsetSyncsTopic(MirrorSourceConnector.java:328)
> at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
> at
> org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
> at
> org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63)
> [...]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TopicAuthorizationException: Authorization
> failed.
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:124)
> ... 11 more
> Caused by: org.apache.kafka.common.errors.TopicAuthorizationException:
> Authorization failed. {code}
>
> h1. Root cause analysis
> The changes introduced by KAFKA-13401 in
> [{{{}MirrorUtils{}}}|https://github.com/apache/kafka/pull/12577/files#diff-fa8f595307a4ade20cc22253a7721828e3b55c96f778e9c4842c978801e0a1a4]
> are supposed to follow the same logic as
> [{{{}TopicAdmin{}}}|https://github.com/apache/kafka/blob/a7e865c0a756504cc7ae6f4eb0772caddc53/connect/runtime/src/main/java/org/apache/kafka/connect/util/TopicAdmin.java#L423]
> according to the contributor's
> [comment|https://github.com/apache/kafka/pull/12577#discussion_r991566108]
> {{TopicAdmin.createOrFindTopics(...)}} and
> {{MirrorUtils.createCompactedTopic(...)}} aren't aligned in terms of allowed
> exceptions
> ||Exception||TopicAdmin||MirrorUtils||
> |TopicExistsException|OK|OK|
> |UnsupportedVersionException|OK|_KO_|
> |ClusterAuthorizationException|OK|_KO_|
> |TopicAuthorizationException|OK|_KO_|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15465) MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
[
https://issues.apache.org/jira/browse/KAFKA-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omnia Ibrahim updated KAFKA-15465:
--
Fix Version/s: 3.6.1
> MM2 not working when its internal topics are pre-created on a cluster that
> disallows topic creation
> ---
>
> Key: KAFKA-15465
> URL: https://issues.apache.org/jira/browse/KAFKA-15465
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.4.1
>Reporter: Ahmed HIBOT
>Assignee: Omnia Ibrahim
>Priority: Major
> Fix For: 3.5.2, 3.7.0, 3.6.1
>
>
> h1. Replication steps
> * Setup a source kafka cluster (alias SOURCE) which doesn't allow topic
> creation to MM2 (therefore it doesn't allow the creation of internal topics)
> * Create MM2 internal topics in the source kafka cluster
> * Setup a target kafka cluster (alias TARGET)
> * Enable one way replication SOURCE->TARGET
> MM2 will attempt to create or find its internal topics on the source cluster
> but it will fail with the following stack trace
> {code:java}
> {"log_timestamp": "2023-09-13T09:39:25.612+", "log_level": "ERROR",
> "process_id": 1, "process_name": "mirror-maker", "thread_id": 1,
> "thread_name": "Scheduler for MirrorSourceConnector-creating upstream
> offset-syncs topic", "action_name":
> "org.apache.kafka.connect.mirror.Scheduler", "log_message": "Scheduler for
> MirrorSourceConnector caught exception in scheduled task: creating upstream
> offset-syncs topic"}
> org.apache.kafka.connect.errors.ConnectException: Error while attempting to
> create/find topic 'mm2-offset-syncs.TARGET.internal'
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:155)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:161)
> at
> org.apache.kafka.connect.mirror.MirrorSourceConnector.createOffsetSyncsTopic(MirrorSourceConnector.java:328)
> at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
> at
> org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
> at
> org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63)
> [...]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TopicAuthorizationException: Authorization
> failed.
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:124)
> ... 11 more
> Caused by: org.apache.kafka.common.errors.TopicAuthorizationException:
> Authorization failed. {code}
>
> h1. Root cause analysis
> The changes introduced by KAFKA-13401 in
> [{{{}MirrorUtils{}}}|https://github.com/apache/kafka/pull/12577/files#diff-fa8f595307a4ade20cc22253a7721828e3b55c96f778e9c4842c978801e0a1a4]
> are supposed to follow the same logic as
> [{{{}TopicAdmin{}}}|https://github.com/apache/kafka/blob/a7e865c0a756504cc7ae6f4eb0772caddc53/connect/runtime/src/main/java/org/apache/kafka/connect/util/TopicAdmin.java#L423]
> according to the contributor's
> [comment|https://github.com/apache/kafka/pull/12577#discussion_r991566108]
> {{TopicAdmin.createOrFindTopics(...)}} and
> {{MirrorUtils.createCompactedTopic(...)}} aren't aligned in terms of allowed
> exceptions
> ||Exception||TopicAdmin||MirrorUtils||
> |TopicExistsException|OK|OK|
> |UnsupportedVersionException|OK|_KO_|
> |ClusterAuthorizationException|OK|_KO_|
> |TopicAuthorizationException|OK|_KO_|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15465) MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
[
https://issues.apache.org/jira/browse/KAFKA-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Satish Duggana updated KAFKA-15465:
---
Fix Version/s: 3.7.0
(was: 3.6.0)
> MM2 not working when its internal topics are pre-created on a cluster that
> disallows topic creation
> ---
>
> Key: KAFKA-15465
> URL: https://issues.apache.org/jira/browse/KAFKA-15465
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.4.1
>Reporter: Ahmed HIBOT
>Assignee: Omnia Ibrahim
>Priority: Major
> Fix For: 3.5.2, 3.7.0
>
>
> h1. Replication steps
> * Setup a source kafka cluster (alias SOURCE) which doesn't allow topic
> creation to MM2 (therefore it doesn't allow the creation of internal topics)
> * Create MM2 internal topics in the source kafka cluster
> * Setup a target kafka cluster (alias TARGET)
> * Enable one way replication SOURCE->TARGET
> MM2 will attempt to create or find its internal topics on the source cluster
> but it will fail with the following stack trace
> {code:java}
> {"log_timestamp": "2023-09-13T09:39:25.612+", "log_level": "ERROR",
> "process_id": 1, "process_name": "mirror-maker", "thread_id": 1,
> "thread_name": "Scheduler for MirrorSourceConnector-creating upstream
> offset-syncs topic", "action_name":
> "org.apache.kafka.connect.mirror.Scheduler", "log_message": "Scheduler for
> MirrorSourceConnector caught exception in scheduled task: creating upstream
> offset-syncs topic"}
> org.apache.kafka.connect.errors.ConnectException: Error while attempting to
> create/find topic 'mm2-offset-syncs.TARGET.internal'
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:155)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:161)
> at
> org.apache.kafka.connect.mirror.MirrorSourceConnector.createOffsetSyncsTopic(MirrorSourceConnector.java:328)
> at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
> at
> org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
> at
> org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63)
> [...]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TopicAuthorizationException: Authorization
> failed.
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:124)
> ... 11 more
> Caused by: org.apache.kafka.common.errors.TopicAuthorizationException:
> Authorization failed. {code}
>
> h1. Root cause analysis
> The changes introduced by KAFKA-13401 in
> [{{{}MirrorUtils{}}}|https://github.com/apache/kafka/pull/12577/files#diff-fa8f595307a4ade20cc22253a7721828e3b55c96f778e9c4842c978801e0a1a4]
> are supposed to follow the same logic as
> [{{{}TopicAdmin{}}}|https://github.com/apache/kafka/blob/a7e865c0a756504cc7ae6f4eb0772caddc53/connect/runtime/src/main/java/org/apache/kafka/connect/util/TopicAdmin.java#L423]
> according to the contributor's
> [comment|https://github.com/apache/kafka/pull/12577#discussion_r991566108]
> {{TopicAdmin.createOrFindTopics(...)}} and
> {{MirrorUtils.createCompactedTopic(...)}} aren't aligned in terms of allowed
> exceptions
> ||Exception||TopicAdmin||MirrorUtils||
> |TopicExistsException|OK|OK|
> |UnsupportedVersionException|OK|_KO_|
> |ClusterAuthorizationException|OK|_KO_|
> |TopicAuthorizationException|OK|_KO_|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15465) MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
[
https://issues.apache.org/jira/browse/KAFKA-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omnia Ibrahim updated KAFKA-15465:
--
Fix Version/s: 3.5.2
(was: 3.7.0)
> MM2 not working when its internal topics are pre-created on a cluster that
> disallows topic creation
> ---
>
> Key: KAFKA-15465
> URL: https://issues.apache.org/jira/browse/KAFKA-15465
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.4.1
>Reporter: Ahmed HIBOT
>Assignee: Omnia Ibrahim
>Priority: Major
> Fix For: 3.6.0, 3.5.2
>
>
> h1. Replication steps
> * Setup a source kafka cluster (alias SOURCE) which doesn't allow topic
> creation to MM2 (therefore it doesn't allow the creation of internal topics)
> * Create MM2 internal topics in the source kafka cluster
> * Setup a target kafka cluster (alias TARGET)
> * Enable one way replication SOURCE->TARGET
> MM2 will attempt to create or find its internal topics on the source cluster
> but it will fail with the following stack trace
> {code:java}
> {"log_timestamp": "2023-09-13T09:39:25.612+", "log_level": "ERROR",
> "process_id": 1, "process_name": "mirror-maker", "thread_id": 1,
> "thread_name": "Scheduler for MirrorSourceConnector-creating upstream
> offset-syncs topic", "action_name":
> "org.apache.kafka.connect.mirror.Scheduler", "log_message": "Scheduler for
> MirrorSourceConnector caught exception in scheduled task: creating upstream
> offset-syncs topic"}
> org.apache.kafka.connect.errors.ConnectException: Error while attempting to
> create/find topic 'mm2-offset-syncs.TARGET.internal'
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:155)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:161)
> at
> org.apache.kafka.connect.mirror.MirrorSourceConnector.createOffsetSyncsTopic(MirrorSourceConnector.java:328)
> at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
> at
> org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
> at
> org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63)
> [...]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TopicAuthorizationException: Authorization
> failed.
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:124)
> ... 11 more
> Caused by: org.apache.kafka.common.errors.TopicAuthorizationException:
> Authorization failed. {code}
>
> h1. Root cause analysis
> The changes introduced by KAFKA-13401 in
> [{{{}MirrorUtils{}}}|https://github.com/apache/kafka/pull/12577/files#diff-fa8f595307a4ade20cc22253a7721828e3b55c96f778e9c4842c978801e0a1a4]
> are supposed to follow the same logic as
> [{{{}TopicAdmin{}}}|https://github.com/apache/kafka/blob/a7e865c0a756504cc7ae6f4eb0772caddc53/connect/runtime/src/main/java/org/apache/kafka/connect/util/TopicAdmin.java#L423]
> according to the contributor's
> [comment|https://github.com/apache/kafka/pull/12577#discussion_r991566108]
> {{TopicAdmin.createOrFindTopics(...)}} and
> {{MirrorUtils.createCompactedTopic(...)}} aren't aligned in terms of allowed
> exceptions
> ||Exception||TopicAdmin||MirrorUtils||
> |TopicExistsException|OK|OK|
> |UnsupportedVersionException|OK|_KO_|
> |ClusterAuthorizationException|OK|_KO_|
> |TopicAuthorizationException|OK|_KO_|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15465) MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
[
https://issues.apache.org/jira/browse/KAFKA-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed HIBOT updated KAFKA-15465:
Description:
h1. Replication steps
* Setup a source kafka cluster (alias SOURCE) which doesn't allow topic
creation to MM2 (therefore it doesn't allow the creation of internal topics)
* Create MM2 internal topics in the source kafka cluster
* Setup a target kafka cluster (alias TARGET)
* Enable one way replication SOURCE->TARGET
MM2 will attempt to create or find its internal topics on the source cluster
but it will fail with the following stack trace
{code:java}
{"log_timestamp": "2023-09-13T09:39:25.612+", "log_level": "ERROR",
"process_id": 1, "process_name": "mirror-maker", "thread_id": 1, "thread_name":
"Scheduler for MirrorSourceConnector-creating upstream offset-syncs topic",
"action_name": "org.apache.kafka.connect.mirror.Scheduler", "log_message":
"Scheduler for MirrorSourceConnector caught exception in scheduled task:
creating upstream offset-syncs topic"}
org.apache.kafka.connect.errors.ConnectException: Error while attempting to
create/find topic 'mm2-offset-syncs.TARGET.internal'
at
org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:155)
at
org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:161)
at
org.apache.kafka.connect.mirror.MirrorSourceConnector.createOffsetSyncsTopic(MirrorSourceConnector.java:328)
at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
at
org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
at
org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63)
[...]
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.TopicAuthorizationException: Authorization
failed.
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at
org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:124)
... 11 more
Caused by: org.apache.kafka.common.errors.TopicAuthorizationException:
Authorization failed. {code}
h1. Root cause analysis
The changes introduced by KAFKA-13401 in
[{{{}MirrorUtils{}}}|https://github.com/apache/kafka/pull/12577/files#diff-fa8f595307a4ade20cc22253a7721828e3b55c96f778e9c4842c978801e0a1a4]
are supposed to follow the same logic as
[{{{}TopicAdmin{}}}|https://github.com/apache/kafka/blob/a7e865c0a756504cc7ae6f4eb0772caddc53/connect/runtime/src/main/java/org/apache/kafka/connect/util/TopicAdmin.java#L423]
according to the contributor's
[comment|https://github.com/apache/kafka/pull/12577#discussion_r991566108]
{{TopicAdmin.createOrFindTopics(...)}} and
{{MirrorUtils.createCompactedTopic(...)}} aren't aligned in terms of allowed
exceptions
||Exception||TopicAdmin||MirrorUtils||
|TopicExistsException|OK|OK|
|UnsupportedVersionException|OK|_KO_|
|ClusterAuthorizationException|OK|_KO_|
|TopicAuthorizationException|OK|_KO_|
> MM2 not working when its internal topics are pre-created on a cluster that
> disallows topic creation
> ---
>
> Key: KAFKA-15465
> URL: https://issues.apache.org/jira/browse/KAFKA-15465
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.4.1
>Reporter: Ahmed HIBOT
>Priority: Major
>
> h1. Replication steps
> * Setup a source kafka cluster (alias SOURCE) which doesn't allow topic
> creation to MM2 (therefore it doesn't allow the creation of internal topics)
> * Create MM2 internal topics in the source kafka cluster
> * Setup a target kafka cluster (alias TARGET)
> * Enable one way replication SOURCE->TARGET
> MM2 will attempt to create or find its internal topics on the source cluster
> but it will fail with the following stack trace
> {code:java}
> {"log_timestamp": "2023-09-13T09:39:25.612+", "log_level": "ERROR",
> "process_id": 1, "process_name": "mirror-maker", "thread_id": 1,
> "thread_name": "Scheduler for MirrorSourceConnector-creating upstream
> offset-syncs topic", "action_name":
> "org.apache.kafka.connect.mirror.Scheduler", "log_message": "Scheduler for
> MirrorSourceConnector caught exception in scheduled task: creating upstream
> offset-syncs topic"}
> org.apache.kafka.connect.errors.ConnectException: Error while attempting to
> create/find topic 'mm2-offset-syncs.TARGET.internal'
> at
> org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:155)
> at
> org.apach
