[jira] [Comment Edited] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184891#comment-17184891 ] Ning Zhang edited comment on KAFKA-10424 at 8/26/20, 3:13 AM: -- [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} After running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:*cleanup.policy=compact* Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} note that `cleanup.policy=compact` is added after MM2 start was (Author: yangguo1220): [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} After running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:cleanup.policy=compact Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} note that `cleanup.policy=compact` is added after MM2 start > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Assignee: Ning Zhang >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184891#comment-17184891 ] Ning Zhang edited comment on KAFKA-10424 at 8/26/20, 3:13 AM: -- [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} After running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:cleanup.policy=compact Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} note that `cleanup.policy=compact` is added after MM2 start was (Author: yangguo1220): [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} After running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:*cleanup.policy=compact* Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} note that `cleanup.policy=compact` is added after MM2 start > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Assignee: Ning Zhang >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184891#comment-17184891 ] Ning Zhang edited comment on KAFKA-10424 at 8/26/20, 3:12 AM: -- [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} After running MM2: {code:java} bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:cleanup.policy=compact Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 {code} note that `cleanup.policy=compact` is added after MM2 start was (Author: yangguo1220): [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: ``` bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 ``` After running MM2: ``` bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:cleanup.policy=compact Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 ``` note that `cleanup.policy=compact` is added after MM2 start > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Assignee: Ning Zhang >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182710#comment-17182710 ] Mikhail Grinfeld edited comment on KAFKA-10424 at 8/24/20, 6:25 AM: I can see the same behaviour with KTable topics generated by KStreams app :( was (Author: grinfeld): I can see the same behaviour with KTable topics generated by KStreams app, but in case I create topic with "*kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1*" the topic not replicated at all. I'll check my configuration, maybe I have some buggy conf in my app :( > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182581#comment-17182581 ] Mikhail Grinfeld edited comment on KAFKA-10424 at 8/23/20, 6:29 AM: I checked again and it never copied cleanup.policy. was (Author: grinfeld): Thx for quick comment. I don't use prefixes, since I migrate data to new servers and only switch between broker.ids on client machines. Maybe, really, the reason is absent of prefixes. I'll try to debug and check it. Thx. > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182572#comment-17182572 ] Ning Zhang edited comment on KAFKA-10424 at 8/23/20, 3:28 AM: -- [~grinfeld] "_schemas" topic is a compacted topic. When it is replicated, make sure its name on the target cluster prefix with , e.g. "primary._schemas" and having your schema registry service on target cluster using the correct topic name. MirrorSourceConnector should periodically sync the topic config (including "cleanup.policy") from source to target (https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L122) so the "_schemas" topic on target cluster should be compacted topic. To debug, check if "_schemas" topic on target cluster is *indeed* compacted topic, especially at the moment when you see the above exception. was (Author: yangguo1220): [~grinfeld] "_schemas" topic is a compacted topic. When it is replicated, make sure its name on the target cluster prefix with , e.g. "primary._schemas" MirrorSourceConnector should periodically sync the topic config (including "cleanup.policy") from source to target (https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L122) so the "_schemas" topic on target cluster should be compacted topic. To debug, check if "_schemas" topic on target cluster is *indeed* compacted topic, especially at the moment when you see the above exception. > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)