[jira] [Commented] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184891#comment-17184891 ] Ning Zhang commented on KAFKA-10424: [~grinfeld] I deployed the latest kafka 2.6 and it seems to me this is not a bug: before running MM2: ``` bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs: Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 ``` After running MM2: ``` bash-4.4# /opt/kafka/bin/kafka-topics.sh --zookeeper zookeeper-service-backup:2181 --describe --topic primary.test Topic:primary.test PartitionCount:3ReplicationFactor:3 Configs:cleanup.policy=compact Topic: primary.test Partition: 0Leader: 1 Replicas: 1,2,0 Isr: 1,2,0 Topic: primary.test Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Topic: primary.test Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,1,2 ``` note that `cleanup.policy=compact` is added after MM2 start > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Assignee: Ning Zhang >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182977#comment-17182977 ] Ning Zhang commented on KAFKA-10424: Actually, I did a quick validation on real deployment, and the first impression is: "cleanup.policy=compact" is not replicated to the remote topic on the target cluster. So it seems a bug to me and need to look into it > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182710#comment-17182710 ] Mikhail Grinfeld commented on KAFKA-10424: -- I can see the same behaviour with KTable topics generated by KStreams app, but in case I create topic with "*kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1*" the topic not replicated at all. I'll check my configuration, maybe I have some buggy conf in my app :( > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182698#comment-17182698 ] Ning Zhang commented on KAFKA-10424: Then can you check if other topic prosperities (if any) are copied over from source to target cluster? > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182581#comment-17182581 ] Mikhail Grinfeld commented on KAFKA-10424: -- Thx for quick comment. I don't use prefixes, since I migrate data to new servers and only switch between broker.ids on client machines. Maybe, really, the reason is absent of prefixes. I'll try to debug and check it. Thx. > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10424) MirrorMaker 2.0 does not replicates topic's "clean.policy"
[ https://issues.apache.org/jira/browse/KAFKA-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182572#comment-17182572 ] Ning Zhang commented on KAFKA-10424: [~grinfeld] "_schemas" topic is a compacted topic. When it is replicated, make sure its name on the target cluster prefix with , e.g. "primary._schemas" MirrorSourceConnector should periodically sync the topic config (including "cleanup.policy") from source to target (https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L122) so the "_schemas" topic on target cluster should be compacted topic. To debug, check if "_schemas" topic on target cluster is *indeed* compacted topic, especially at the moment when you see the above exception. > MirrorMaker 2.0 does not replicates topic's "clean.policy" > -- > > Key: KAFKA-10424 > URL: https://issues.apache.org/jira/browse/KAFKA-10424 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.5.0 >Reporter: Mikhail Grinfeld >Priority: Major > > I needed to replicate schema-registry "_schemas" topic. > data was replicated successfully and everything looked good, but new > schema-registry started with warning that replicated topic's cleanup.policy > is not "compact" -- This message was sent by Atlassian Jira (v8.3.4#803005)
