[jira] [Updated] (KAFKA-15262) MirrorHeartbeatConnector is not working as documented
[ https://issues.apache.org/jira/browse/KAFKA-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindranath Kakarla updated KAFKA-15262: - Description: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. However, this is not happening right now. To the contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? *Reference:* _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] was: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > MirrorHeartbeatConnector is not working as documented > - > > Key: KAFKA-15262 > URL: https://issues.apache.org/jira/browse/KAFKA-15262 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.8.0, 3.4.0, 3.5.0 >Reporter: Ravindranath Kakarla >Priority: Major > > As per the MM2 > [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] > the MirrorHeartbeatConnector should emit pings to heartbeat topic on the > source cluster which then gets replicated to the target cluster. This can be > used to demonstrate that MM2 is replicating the data. > However, this is not happening right now. To the contrary, the > MirrorHeartbeatConnector is producing heartbeat pings to target cluster > instead of source. This is not much useful as it won't help detect problems > connecting to source cluster or with the data replication. > Is my understanding of the MirrorHeartbeatConnector accurate? > *Reference:* > _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated > to demonstrate connectivity through the connectors. Downstream consumers can > use this topic to verify that a) the connector is running and b) the > corresponding source cluster is available. Heartbeats will get propagated by > source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat > are possible._ > [Code > Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15262) MirrorHeartbeatConnector is not working as documented
[ https://issues.apache.org/jira/browse/KAFKA-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindranath Kakarla updated KAFKA-15262: - Description: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] was: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > MirrorHeartbeatConnector is not working as documented > - > > Key: KAFKA-15262 > URL: https://issues.apache.org/jira/browse/KAFKA-15262 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.8.0, 3.4.0, 3.5.0 >Reporter: Ravindranath Kakarla >Priority: Major > > As per the MM2 > [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] > the MirrorHeartbeatConnector should emit pings to heartbeat topic on the > source cluster which then gets replicated to the target cluster. This can be > used to demonstrate that MM2 is replicating the data. > h2. _Internal Topics_ > _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated > to demonstrate connectivity through the connectors. Downstream consumers can > use this topic to verify that a) the connector is running and b) the > corresponding source cluster is available. Heartbeats will get propagated by > source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat > are possible._ > However, this is not happening right now. To contrary, the > MirrorHeartbeatConnector is producing heartbeat pings to target cluster > instead of source. This is not much useful as it won't help detect problems > connecting to source cluster or with the data replication. > > Is my understanding of the MirrorHeartbeatConnector accurate? > [Code > Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15262) MirrorHeartbeatConnector is not working as documented
[ https://issues.apache.org/jira/browse/KAFKA-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindranath Kakarla updated KAFKA-15262: - Description: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] was: As per the MM2 [KIP-382 | [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > MirrorHeartbeatConnector is not working as documented > - > > Key: KAFKA-15262 > URL: https://issues.apache.org/jira/browse/KAFKA-15262 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.8.0, 3.4.0, 3.5.0 >Reporter: Ravindranath Kakarla >Priority: Major > > As per the MM2 > [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] > the MirrorHeartbeatConnector should emit pings to heartbeat topic on the > source cluster which then gets replicated to the target cluster. This can be > used to demonstrate that MM2 is replicating the data. > h2. _Internal Topics_ > _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated > to demonstrate connectivity through the connectors. Downstream consumers can > use this topic to verify that a) the connector is running and b) the > corresponding source cluster is available. Heartbeats will get propagated by > source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat > are possible._ > > > However, this is not happening right now. To contrary, the > MirrorHeartbeatConnector is producing heartbeat pings to target cluster > instead of source. This is not much useful as it won't help detect problems > connecting to source cluster or with the data replication. > > Is my understanding of the MirrorHeartbeatConnector accurate? > [Code > Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15262) MirrorHeartbeatConnector is not working as documented
[ https://issues.apache.org/jira/browse/KAFKA-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindranath Kakarla updated KAFKA-15262: - Description: As per the MM2 [KIP-382 | [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] was: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > MirrorHeartbeatConnector is not working as documented > - > > Key: KAFKA-15262 > URL: https://issues.apache.org/jira/browse/KAFKA-15262 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.8.0, 3.4.0, 3.5.0 >Reporter: Ravindranath Kakarla >Priority: Major > > As per the MM2 [KIP-382 | > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] > the MirrorHeartbeatConnector should emit pings to heartbeat topic on the > source cluster which then gets replicated to the target cluster. This can be > used to demonstrate that MM2 is replicating the data. > h2. _Internal Topics_ > _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated > to demonstrate connectivity through the connectors. Downstream consumers can > use this topic to verify that a) the connector is running and b) the > corresponding source cluster is available. Heartbeats will get propagated by > source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat > are possible._ > > > However, this is not happening right now. To contrary, the > MirrorHeartbeatConnector is producing heartbeat pings to target cluster > instead of source. This is not much useful as it won't help detect problems > connecting to source cluster or with the data replication. > > Is my understanding of the MirrorHeartbeatConnector accurate? > [Code > Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15262) MirrorHeartbeatConnector is not working as documented
[ https://issues.apache.org/jira/browse/KAFKA-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindranath Kakarla updated KAFKA-15262: - Description: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. h2. _Internal Topics_ _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible._ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] was: As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0],] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. """ h2. Internal Topics MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible. """ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > MirrorHeartbeatConnector is not working as documented > - > > Key: KAFKA-15262 > URL: https://issues.apache.org/jira/browse/KAFKA-15262 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 2.8.0, 3.4.0, 3.5.0 >Reporter: Ravindranath Kakarla >Priority: Major > > As per the MM2 > [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]] > the MirrorHeartbeatConnector should emit pings to heartbeat topic on the > source cluster which then gets replicated to the target cluster. This can be > used to demonstrate that MM2 is replicating the data. > h2. _Internal Topics_ > _MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated > to demonstrate connectivity through the connectors. Downstream consumers can > use this topic to verify that a) the connector is running and b) the > corresponding source cluster is available. Heartbeats will get propagated by > source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat > are possible._ > > > However, this is not happening right now. To contrary, the > MirrorHeartbeatConnector is producing heartbeat pings to target cluster > instead of source. This is not much useful as it won't help detect problems > connecting to source cluster or with the data replication. > > Is my understanding of the MirrorHeartbeatConnector accurate? > [Code > Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15262) MirrorHeartbeatConnector is not working as documented
Ravindranath Kakarla created KAFKA-15262: Summary: MirrorHeartbeatConnector is not working as documented Key: KAFKA-15262 URL: https://issues.apache.org/jira/browse/KAFKA-15262 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 3.5.0, 3.4.0, 2.8.0 Reporter: Ravindranath Kakarla As per the MM2 [KIP-382|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0],] the MirrorHeartbeatConnector should emit pings to heartbeat topic on the source cluster which then gets replicated to the target cluster. This can be used to demonstrate that MM2 is replicating the data. """ h2. Internal Topics MM2 emits a *heartbeat* *topic* in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible. """ However, this is not happening right now. To contrary, the MirrorHeartbeatConnector is producing heartbeat pings to target cluster instead of source. This is not much useful as it won't help detect problems connecting to source cluster or with the data replication. Is my understanding of the MirrorHeartbeatConnector accurate? [Code Ref|https://github.com/apache/kafka/blob/ed44bcd71b3b9926c474033882eaa6c1cf35cfa4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorHeartbeatTask.java#L65] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-13988) Mirrormaker 2 auto.offset.reset=latest not working
[
https://issues.apache.org/jira/browse/KAFKA-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736629#comment-17736629
]
Ravindranath Kakarla commented on KAFKA-13988:
--
Created a PR, https://github.com/apache/kafka/pull/13905
> Mirrormaker 2 auto.offset.reset=latest not working
> --
>
> Key: KAFKA-13988
> URL: https://issues.apache.org/jira/browse/KAFKA-13988
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 3.2.0
> Environment: Source Kafka cluster running on Ubuntu 20
> Source Kafka cluster Kafka v0.10
> Target Kafka cluster running in AWS MSK
> Target Kafka cluster Kafka v2.6.2
> Mirrormaker version 3.2.0 running on Ubuntu 20.
>Reporter: Daniel Florek
>Assignee: Justinwins
>Priority: Major
> Fix For: 3.2.0
>
>
> Hi.
> I have problem setting up mirroring with MM2 from latest offset between 2
> clusters. In logs I can see that Consumer that is consuming topics has
> auto.offset.reset property set to latest. But still topics are read from
> offset 0. I am using following configuration:
>
> {code:java}
> clusters = A, B
> A.bootstrap.servers = broker-01A:9092
> B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092
> replication.policy.class =
> org.apache.kafka.connect.mirror.IdentityReplicationPolicy
> #Enable replication between clusters and define topics which should be
> replicated
> A->B.enabled = true
> A->B.topics = .*
> A->B.replication.factor=3
> A->B.emit.heartbeats.enabled = true
> A->B.emit.checkpoints.enabled = true
> auto.offset.reset=latest
> consumer.auto.offset.reset=latest
> A.consumer.auto.offset.reset=latest
> B.consumer.auto.offset.reset=latest
> refresh.topics.enabled=true
> heartbeats.topic.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> config.storage.replication.factor = 1
> offset.storage.replication.factor = 1
> status.storage.replication.factor = 1 {code}
> I am using Kafka 3.2.0 for Mirrormaker 2. Source kafka cluster is 1 broker
> running on EC2 instance in AWS (quite an old version I think 0.10). Target
> kafka cluster contains 3 brokers running in AWS MSK (version 2.6.2).
> Could you point me what I am doing wrong? Or is this possibly a bug?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-13988) Mirrormaker 2 auto.offset.reset=latest not working
[
https://issues.apache.org/jira/browse/KAFKA-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindranath Kakarla reassigned KAFKA-13988:
Assignee: Ravindranath Kakarla (was: Justinwins)
> Mirrormaker 2 auto.offset.reset=latest not working
> --
>
> Key: KAFKA-13988
> URL: https://issues.apache.org/jira/browse/KAFKA-13988
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 3.2.0
> Environment: Source Kafka cluster running on Ubuntu 20
> Source Kafka cluster Kafka v0.10
> Target Kafka cluster running in AWS MSK
> Target Kafka cluster Kafka v2.6.2
> Mirrormaker version 3.2.0 running on Ubuntu 20.
>Reporter: Daniel Florek
>Assignee: Ravindranath Kakarla
>Priority: Major
> Fix For: 3.2.0
>
>
> Hi.
> I have problem setting up mirroring with MM2 from latest offset between 2
> clusters. In logs I can see that Consumer that is consuming topics has
> auto.offset.reset property set to latest. But still topics are read from
> offset 0. I am using following configuration:
>
> {code:java}
> clusters = A, B
> A.bootstrap.servers = broker-01A:9092
> B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092
> replication.policy.class =
> org.apache.kafka.connect.mirror.IdentityReplicationPolicy
> #Enable replication between clusters and define topics which should be
> replicated
> A->B.enabled = true
> A->B.topics = .*
> A->B.replication.factor=3
> A->B.emit.heartbeats.enabled = true
> A->B.emit.checkpoints.enabled = true
> auto.offset.reset=latest
> consumer.auto.offset.reset=latest
> A.consumer.auto.offset.reset=latest
> B.consumer.auto.offset.reset=latest
> refresh.topics.enabled=true
> heartbeats.topic.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> config.storage.replication.factor = 1
> offset.storage.replication.factor = 1
> status.storage.replication.factor = 1 {code}
> I am using Kafka 3.2.0 for Mirrormaker 2. Source kafka cluster is 1 broker
> running on EC2 instance in AWS (quite an old version I think 0.10). Target
> kafka cluster contains 3 brokers running in AWS MSK (version 2.6.2).
> Could you point me what I am doing wrong? Or is this possibly a bug?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-13988) Mirrormaker 2 auto.offset.reset=latest not working
[
https://issues.apache.org/jira/browse/KAFKA-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17732778#comment-17732778
]
Ravindranath Kakarla commented on KAFKA-13988:
--
Does the issue have anything to do with source cluster being old version
(0.10)? Did someone face this issue with latest versions of Kafka?
> Mirrormaker 2 auto.offset.reset=latest not working
> --
>
> Key: KAFKA-13988
> URL: https://issues.apache.org/jira/browse/KAFKA-13988
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 3.2.0
> Environment: Source Kafka cluster running on Ubuntu 20
> Source Kafka cluster Kafka v0.10
> Target Kafka cluster running in AWS MSK
> Target Kafka cluster Kafka v2.6.2
> Mirrormaker version 3.2.0 running on Ubuntu 20.
>Reporter: Daniel Florek
>Assignee: Justinwins
>Priority: Major
> Fix For: 3.2.0
>
>
> Hi.
> I have problem setting up mirroring with MM2 from latest offset between 2
> clusters. In logs I can see that Consumer that is consuming topics has
> auto.offset.reset property set to latest. But still topics are read from
> offset 0. I am using following configuration:
>
> {code:java}
> clusters = A, B
> A.bootstrap.servers = broker-01A:9092
> B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092
> replication.policy.class =
> org.apache.kafka.connect.mirror.IdentityReplicationPolicy
> #Enable replication between clusters and define topics which should be
> replicated
> A->B.enabled = true
> A->B.topics = .*
> A->B.replication.factor=3
> A->B.emit.heartbeats.enabled = true
> A->B.emit.checkpoints.enabled = true
> auto.offset.reset=latest
> consumer.auto.offset.reset=latest
> A.consumer.auto.offset.reset=latest
> B.consumer.auto.offset.reset=latest
> refresh.topics.enabled=true
> heartbeats.topic.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> config.storage.replication.factor = 1
> offset.storage.replication.factor = 1
> status.storage.replication.factor = 1 {code}
> I am using Kafka 3.2.0 for Mirrormaker 2. Source kafka cluster is 1 broker
> running on EC2 instance in AWS (quite an old version I think 0.10). Target
> kafka cluster contains 3 brokers running in AWS MSK (version 2.6.2).
> Could you point me what I am doing wrong? Or is this possibly a bug?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14952) Publish metrics when source connector fails to poll data
[
https://issues.apache.org/jira/browse/KAFKA-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindranath Kakarla updated KAFKA-14952:
-
Description:
Currently, there is no metric in Kafka Connect to track when a source connector
fails to poll data from the source. This information would be useful to
operators and developers to visualize, monitor and alert when the connector
fails to poll records from the source.
Existing metrics like *kafka_producer_producer_metrics_record_error_total* and
*kafka_connect_task_error_metrics_total_record_failures* only cover failures
when producing data to the Kafka cluster but not when the source task fails
with a retryable exception or ConnectException.
Polling from source can fail due to unavailability of the source system or
errors with the connect configuration. Currently, this cannot be monitored
directly using metrics and instead operators have to rely on log diving which
is not consistent with how other metrics are monitored.
I propose adding new metrics to Kafka Connect,
"{_}source-record-poll-error-total{_}" and
"{_}source-record-poll-error-rate{_}" that can be used to monitor failures
during polling.
*source-record-poll-error-total* - The total number of times a source connector
failed to poll data from the source. This will include both retryable and
non-retryable exceptions.
*source-record-poll-error-rate* - The rate of above failures per unit of time.
These metrics would be tracked at the connector level and could be exposed
through the JMX along with the other metrics.
I am willing to submit a PR if this looks good, sample implementation code
below,
{code:java}
//AbstractWorkerSourceTask.java
protected List poll() throws InterruptedException {
try {
return task.poll();
} catch (RetriableException |
org.apache.kafka.common.errors.RetriableException e) {
log.warn("{} failed to poll records from SourceTask. Will retry
operation.", this, e);
sourceTaskMetricsGroup.recordPollError();
// Do nothing. Let the framework poll whenever it's ready.
return null;
} catch (Throwable e) {
sourceTaskMetricsGroup.recordPollError();
throw e;
}
} {code}
[Reference|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L460]
was:
Currently, there is no metric in Kafka Connect to track when a source connector
fails to poll data from the source. This information would be useful to
operators and developers to visualize, monitor and alert when the connector
fails to poll records from the source.
Existing metrics like *kafka_producer_producer_metrics_record_error_total* and
*kafka_connect_task_error_metrics_total_record_failures* only cover failures
when producing data to the Kafka cluster but not when the source task fails
with a retryable exception or ConnectException.
Polling from source can fail due to unavailability of the source system or
errors with the connect configuration. Currently, this cannot be monitored
directly using metrics and instead operators have to rely on log diving which
is not consistent with how other metrics are monitored.
I propose adding new metrics to Kafka Connect, "source-record-poll-error-total"
and "source-record-poll-error-rate" that can be used to monitor failures during
polling.
_*source-record-poll-error-total*_ - The total number of times a source
connector failed to poll data from the source. This will include both retryable
and non-retryable exceptions.
_*source-record-poll-error-rate*_ - The rate of above failures per unit of time.
These metrics would be tracked at the connector level and could be exposed
through the JMX along with the other metrics.
I am willing to submit a PR if this looks good, sample implementation code
below,
{code:java}
//AbstractWorkerSourceTask.java
protected List poll() throws InterruptedException {
try {
return task.poll();
} catch (RetriableException |
org.apache.kafka.common.errors.RetriableException e) {
log.warn("{} failed to poll records from SourceTask. Will retry
operation.", this, e);
sourceTaskMetricsGroup.recordPollError();
// Do nothing. Let the framework poll whenever it's ready.
return null;
} catch (Throwable e) {
sourceTaskMetricsGroup.recordPollError();
throw e;
}
} {code}
[Reference|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L460]
> Publish metrics when source connector fails to poll data
>
>
> Key: KAFKA-14952
> URL: https://issues.apache.org/jira/browse/KAFKA-14952
> Project: Kafka
> Issue
[jira] [Updated] (KAFKA-14952) Publish metrics when source connector fails to poll data
[
https://issues.apache.org/jira/browse/KAFKA-14952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindranath Kakarla updated KAFKA-14952:
-
Description:
Currently, there is no metric in Kafka Connect to track when a source connector
fails to poll data from the source. This information would be useful to
operators and developers to visualize, monitor and alert when the connector
fails to poll records from the source.
Existing metrics like *kafka_producer_producer_metrics_record_error_total* and
*kafka_connect_task_error_metrics_total_record_failures* only cover failures
when producing data to the Kafka cluster but not when the source task fails
with a retryable exception or ConnectException.
Polling from source can fail due to unavailability of the source system or
errors with the connect configuration. Currently, this cannot be monitored
directly using metrics and instead operators have to rely on log diving which
is not consistent with how other metrics are monitored.
I propose adding new metrics to Kafka Connect, "source-record-poll-error-total"
and "source-record-poll-error-rate" that can be used to monitor failures during
polling.
_*source-record-poll-error-total*_ - The total number of times a source
connector failed to poll data from the source. This will include both retryable
and non-retryable exceptions.
_*source-record-poll-error-rate*_ - The rate of above failures per unit of time.
These metrics would be tracked at the connector level and could be exposed
through the JMX along with the other metrics.
I am willing to submit a PR if this looks good, sample implementation code
below,
{code:java}
//AbstractWorkerSourceTask.java
protected List poll() throws InterruptedException {
try {
return task.poll();
} catch (RetriableException |
org.apache.kafka.common.errors.RetriableException e) {
log.warn("{} failed to poll records from SourceTask. Will retry
operation.", this, e);
sourceTaskMetricsGroup.recordPollError();
// Do nothing. Let the framework poll whenever it's ready.
return null;
} catch (Throwable e) {
sourceTaskMetricsGroup.recordPollError();
throw e;
}
} {code}
[Reference|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L460]
was:
Currently, there is no metric in Kafka Connect to track when a source connector
fails to poll data from the source. This information would be useful to
operators and developers to visualize, monitor and alert when the connector
fails to poll records from the source.
Existing metrics like `kafka_producer_producer_metrics_record_error_total` and
`kafka_connect_task_error_metrics_total_record_failures` only cover failures
when producing data to the Kafka cluster but not when the source task fails
with a retryable exception or ConnectException.
Polling from source can fail due to unavailability of the source system or
errors with the connect configuration. Currently, this cannot be monitored
directly using metrics and instead operators have to rely on log diving which
is not consistent with how other metrics are monitored.
I propose adding new metrics to Kafka Connect, "source-record-poll-error-total"
and "source-record-poll-error-rate" that can be used to monitor failures during
polling.
`source-record-poll-error-total` - The total number of times a source connector
failed to poll data from the source. This will include both retryable and
non-retryable exceptions.
`source-record-poll-error-rate` - The rate of above failures per unit of time.
These metrics would be tracked at the connector level and could be exposed
through the JMX along with the other metrics.
I am willing to submit a PR if this looks good, sample implementation code
below,
{code:java}
//AbstractWorkerSourceTask.java
protected List poll() throws InterruptedException {
try {
return task.poll();
} catch (RetriableException |
org.apache.kafka.common.errors.RetriableException e) {
log.warn("{} failed to poll records from SourceTask. Will retry
operation.", this, e);
sourceTaskMetricsGroup.recordPollError();
// Do nothing. Let the framework poll whenever it's ready.
return null;
} catch (Throwable e) {
sourceTaskMetricsGroup.recordPollError();
throw e;
}
} {code}
> Publish metrics when source connector fails to poll data
>
>
> Key: KAFKA-14952
> URL: https://issues.apache.org/jira/browse/KAFKA-14952
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 3.3.2
>Reporter: Ravindranath Kakarla
>Priority: Minor
>
[jira] [Created] (KAFKA-14952) Publish metrics when source connector fails to poll data
Ravindranath Kakarla created KAFKA-14952:
Summary: Publish metrics when source connector fails to poll data
Key: KAFKA-14952
URL: https://issues.apache.org/jira/browse/KAFKA-14952
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Affects Versions: 3.3.2
Reporter: Ravindranath Kakarla
Currently, there is no metric in Kafka Connect to track when a source connector
fails to poll data from the source. This information would be useful to
operators and developers to visualize, monitor and alert when the connector
fails to poll records from the source.
Existing metrics like `kafka_producer_producer_metrics_record_error_total` and
`kafka_connect_task_error_metrics_total_record_failures` only cover failures
when producing data to the Kafka cluster but not when the source task fails
with a retryable exception or ConnectException.
Polling from source can fail due to unavailability of the source system or
errors with the connect configuration. Currently, this cannot be monitored
directly using metrics and instead operators have to rely on log diving which
is not consistent with how other metrics are monitored.
I propose adding new metrics to Kafka Connect, "source-record-poll-error-total"
and "source-record-poll-error-rate" that can be used to monitor failures during
polling.
`source-record-poll-error-total` - The total number of times a source connector
failed to poll data from the source. This will include both retryable and
non-retryable exceptions.
`source-record-poll-error-rate` - The rate of above failures per unit of time.
These metrics would be tracked at the connector level and could be exposed
through the JMX along with the other metrics.
I am willing to submit a PR if this looks good, sample implementation code
below,
{code:java}
//AbstractWorkerSourceTask.java
protected List poll() throws InterruptedException {
try {
return task.poll();
} catch (RetriableException |
org.apache.kafka.common.errors.RetriableException e) {
log.warn("{} failed to poll records from SourceTask. Will retry
operation.", this, e);
sourceTaskMetricsGroup.recordPollError();
// Do nothing. Let the framework poll whenever it's ready.
return null;
} catch (Throwable e) {
sourceTaskMetricsGroup.recordPollError();
throw e;
}
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-10715) Support Kafka connect converter for AVRO
[ https://issues.apache.org/jira/browse/KAFKA-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindranath Kakarla updated KAFKA-10715: - Remaining Estimate: 72h (was: 336h) Original Estimate: 72h (was: 336h) > Support Kafka connect converter for AVRO > > > Key: KAFKA-10715 > URL: https://issues.apache.org/jira/browse/KAFKA-10715 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ravindranath Kakarla >Priority: Minor > Original Estimate: 72h > Remaining Estimate: 72h > > I want to add support for Avro data format converter to Kafka Connect. Right > now, Kafka connect supports [JSON > converter|[https://github.com/apache/kafka/tree/trunk/connect].] Since, Avro > is a commonly used data format with Kafka, it will be great to have support > for it. > > Confluent Schema Registry libraries have > [support|https://github.com/confluentinc/schema-registry/blob/master/avro-converter/src/main/java/io/confluent/connect/avro/AvroConverter.java] > for it. The code seems to be pretty generic and can be used directly with > Kafka connect without schema registry. They are also licensed under Apache > 2.0. > > Can they be copied to this repository and made available for all users of > Kafka Connect? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10715) Support Kafka connect converter for AVRO
[ https://issues.apache.org/jira/browse/KAFKA-10715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245501#comment-17245501 ] Ravindranath Kakarla commented on KAFKA-10715: -- Can I contribute this change? > Support Kafka connect converter for AVRO > > > Key: KAFKA-10715 > URL: https://issues.apache.org/jira/browse/KAFKA-10715 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Ravindranath Kakarla >Priority: Minor > Original Estimate: 336h > Remaining Estimate: 336h > > I want to add support for Avro data format converter to Kafka Connect. Right > now, Kafka connect supports [JSON > converter|[https://github.com/apache/kafka/tree/trunk/connect].] Since, Avro > is a commonly used data format with Kafka, it will be great to have support > for it. > > Confluent Schema Registry libraries have > [support|https://github.com/confluentinc/schema-registry/blob/master/avro-converter/src/main/java/io/confluent/connect/avro/AvroConverter.java] > for it. The code seems to be pretty generic and can be used directly with > Kafka connect without schema registry. They are also licensed under Apache > 2.0. > > Can they be copied to this repository and made available for all users of > Kafka Connect? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10715) Support Kafka connect converter for AVRO
Ravindranath Kakarla created KAFKA-10715: Summary: Support Kafka connect converter for AVRO Key: KAFKA-10715 URL: https://issues.apache.org/jira/browse/KAFKA-10715 Project: Kafka Issue Type: New Feature Components: KafkaConnect Reporter: Ravindranath Kakarla I want to add support for Avro data format converter to Kafka Connect. Right now, Kafka connect supports [JSON converter|[https://github.com/apache/kafka/tree/trunk/connect].] Since, Avro is a commonly used data format with Kafka, it will be great to have support for it. Confluent Schema Registry libraries have [support|https://github.com/confluentinc/schema-registry/blob/master/avro-converter/src/main/java/io/confluent/connect/avro/AvroConverter.java] for it. The code seems to be pretty generic and can be used directly with Kafka connect without schema registry. They are also licensed under Apache 2.0. Can they be copied to this repository and made available for all users of Kafka Connect? -- This message was sent by Atlassian Jira (v8.3.4#803005)
