I just figured out why there are so many Disconnection exceptions .. the
podIP keeps changing in GCP, hence when MirrorMaker tries to access the pod
using the older IP, the pod is not found .. hence the DisconnectException
However, it is still not clear why the MirrorMaker is not transferring data
from src topic to target topic
On Thu, Feb 23, 2023 at 3:04 PM karan alang <karan.al...@gmail.com> wrote:
> Logs with the Disconnect Exception (when i set the loglevel to - INFO) :
>
>
> ```
>
> [SourceTaskOffsetCommitter-1]
>
> 2023-02-23 22:54:13,546 INFO [Consumer clientId=consumer-null-4,
> groupId=null] Error sending fetch request (sessionId=1509420532, epoch=492)
> to node 2: (org.apache.kafka.clients.FetchSessionHandler)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> org.apache.kafka.common.errors.DisconnectException
>
> 2023-02-23 22:54:13,711 WARN [Consumer clientId=consumer-null-4,
> groupId=null] Connection to node 2
> (nossl-w-kafka-2.nossl-w-kafka-brokers.kafka.svc/10.98.129.75:9092) could
> not be established. Broker may not be available.
> (org.apache.kafka.clients.NetworkClient)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:54:13,711 INFO [Consumer clientId=consumer-null-4,
> groupId=null] Error sending fetch request (sessionId=1509420532,
> epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> org.apache.kafka.common.errors.DisconnectException
>
> 2023-02-23 22:54:24,744 INFO
> WorkerSourceTask{id=my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0}
> flushing 0 outstanding messages for offset commit
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [SourceTaskOffsetCommitter-1]
>
> 2023-02-23 22:54:37,029 INFO [Consumer clientId=consumer-null-4,
> groupId=null] Error sending fetch request (sessionId=1509420532,
> epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> org.apache.kafka.common.errors.DisconnectException
>
> 2023-02-23 22:55:06,431 INFO
> WorkerSourceTask{id=my-source-cluster-west->my-target-cluster-east.MirrorHeartbeatConnector-0}
> flushing 0 outstanding messages for offset commit
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [SourceTaskOffsetCommitter-1]
>
> 2023-02-23 22:55:07,449 INFO [Consumer clientId=consumer-null-4,
> groupId=null] Error sending fetch request (sessionId=1509420532,
> epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> org.apache.kafka.common.errors.TimeoutException: Failed to send request
> after 30000 ms.
>
> 2023-02-23 22:55:10,029 INFO [Consumer clientId=consumer-null-4,
> groupId=null] Error sending fetch request (sessionId=1509420532,
> epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> org.apache.kafka.common.errors.DisconnectException
>
> 2023-02-23 22:55:24,745 INFO
> WorkerSourceTask{id=my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0}
> flushing 0 outstanding messages for offset commit
> (org.apache.kafka.connect.runtime.WorkerSourceTask)
> [SourceTaskOffsetCommitter-1]
>
> ```
>
> On Thu, Feb 23, 2023 at 2:12 PM karan alang <karan.al...@gmail.com> wrote:
>
>> Hello All,
>>
>> Anyone has installed KafkaMirrorMaker2 on GKE ?
>> Need some help to debug/resolve issues that I've been having.
>>
>> I've 2 clusters on GKE with strimzi kafka installed.
>> cluster1(nossl-e) on use-east1, cluster2(nossl-e) on us-west1.
>>
>> Both the clusters are Auto-pilot cluster (mentioning this, thought it
>> might not be relevant for the issues being faced)
>>
>> source cluster -> nossl-w
>> target cluster -> nossl-e
>>
>> Currently(to get MM2 working), SSL & authorization are not enabled.
>>
>> KafkaMirrorMaker2 is installed on source cluster i.e. nossl-w
>>
>> Here is the yaml :
>> ```
>> apiVersion: kafka.strimzi.io/v1beta2
>> kind: KafkaMirrorMaker2
>> metadata:
>> name: my-mirror-maker-2
>> spec:
>> version: 3.0.0
>> replicas: 1
>> connectCluster: "my-target-cluster-east"
>> clusters:
>> - alias: "my-source-cluster-west"
>> bootstrapServers: nossl-w-kafka-bootstrap:9092
>> - alias: "my-target-cluster-east"
>> bootstrapServers: xx.xx.xx.xx:9094
>> config:
>> # -1 means it will use the default replication factor configured in
>> the broker
>> config.storage.replication.factor: -1
>> offset.storage.replication.factor: -1
>> status.storage.replication.factor: -1
>> mirrors:
>> - sourceCluster: "my-source-cluster-west"
>> targetCluster: "my-target-cluster-east"
>> sourceConnector:
>> config:
>> replication.factor: 1
>> offset-syncs.topic.replication.factor: 1
>> sync.topic.acls.enabled: "false"
>> heartbeatConnector:
>> config:
>> heartbeats.topic.replication.factor: 1
>> checkpointConnector:
>> config:
>> checkpoints.topic.replication.factor: 1
>> topicsPattern: ".*"
>> groupsPattern: ".*"
>>
>> ```
>>
>> While the pods & the svc come up, MirrorMaker is not transferring data
>> from source to target topics. (mmtest - in my case, that is the only topic
>> created on both the clusters)
>>
>> Here are the logs from the mirrormaker pod :
>>
>>
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Requesting metadata update for partition
>> mirrormaker2-cluster-offsets-7 due to error LEADER_NOT_AVAILABLE
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-offsets-21 from 226 to epoch 226 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-status-0 from 225 to epoch 225 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Requesting metadata update for partition
>> mirrormaker2-cluster-status-0 due to error LEADER_NOT_AVAILABLE
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-status-1 from 226 to epoch 226 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-status-4 from 226 to epoch 226 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-status-2 from 226 to epoch 226 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-status-3 from 225 to epoch 225 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Requesting metadata update for partition
>> mirrormaker2-cluster-status-3 due to error LEADER_NOT_AVAILABLE
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition mmtest-0 from 460 to
>> epoch 460 from new metadata (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition mmtest-2 from 416 to
>> epoch 416 from new metadata (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition mmtest-1 from 433 to
>> epoch 433 from new metadata (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updating last seen epoch for partition
>> mirrormaker2-cluster-configs-0 from 226 to epoch 226 from new metadata
>> (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Updated cluster metadata updateVersion 1597 to
>> MetadataCache{clusterId='yoD3YJaKTz6dvaVT2MfiOw',
>> nodes={1=nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack:
>> null), 2=nossl-w-kafka-2.nossl-w-kafka-brokers.kafka.svc:9092 (id: 2 rack:
>> null)}, partitions=[PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-0, leader=Optional[1],
>> leaderEpoch=Optional[226], replicas=1, isr=1, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-status-3, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-2,
>> leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-status-1, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-4, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mmtest-0, leader=Optional[2],
>> leaderEpoch=Optional[460], replicas=2,1,0, isr=1,2, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-6,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE, partition=mmtest-2,
>> leader=Optional[2], leaderEpoch=Optional[416], replicas=0,2,1, isr=1,2,
>> offlineReplicas=0), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-8, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-10, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-12,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-14, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-15,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-17, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-19, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-21,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-23, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-1, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-status-4,
>> leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-3, leader=Optional[1],
>> leaderEpoch=Optional[226], replicas=1, isr=1, offlineReplicas=),
>> PartitionMetadata(error=NONE, partition=mmtest-1, leader=Optional[1],
>> leaderEpoch=Optional[433], replicas=1,0,2, isr=1,2, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-status-2,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-5, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-status-0, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-7, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-9,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-offsets-11, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-13, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-16, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-18,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=), PartitionMetadata(error=NONE,
>> partition=mirrormaker2-cluster-configs-0, leader=Optional[2],
>> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-20,
>> leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
>> offlineReplicas=), PartitionMetadata(error=LEADER_NOT_AVAILABLE,
>> partition=mirrormaker2-cluster-offsets-22, leader=Optional.empty,
>> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
>> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-24,
>> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
>> offlineReplicas=)],
>> controller=nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1
>> rack: null)} (org.apache.kafka.clients.Metadata)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>>
>> 2023-02-23 22:05:54,666 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Received FETCH response from node 1 for request with header
>> RequestHeader(apiKey=FETCH, apiVersion=12,
>> clientId=consumer-mirrormaker2-cluster-1, correlationId=12072):
>> FetchResponseData(throttleTimeMs=0, errorCode=0, sessionId=1476981493,
>> responses=[]) (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,666 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Node 1 sent an incremental fetch response with throttleTimeMs = 0 for
>> session 1476981493 with 0 response partition(s), 8 implied partition(s)
>> (org.apache.kafka.clients.FetchSessionHandler) [KafkaBasedLog Work Thread -
>> mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-2 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-8 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-17 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-23 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-5 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-11 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-20 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-14 at position FetchPosition{offset=2,
>> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
>> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
>> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Built incremental fetch (sessionId=1476981493, epoch=4013) for node 1.
>> Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of
>> 8 partition(s) (org.apache.kafka.clients.FetchSessionHandler)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(),
>> implied=(mirrormaker2-cluster-offsets-2, mirrormaker2-cluster-offsets-8,
>> mirrormaker2-cluster-offsets-14, mirrormaker2-cluster-offsets-17,
>> mirrormaker2-cluster-offsets-23, mirrormaker2-cluster-offsets-5,
>> mirrormaker2-cluster-offsets-11, mirrormaker2-cluster-offsets-20)) to
>> broker 34.75.2.112:9094 (id: 1 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,667 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Sending FETCH request with header RequestHeader(apiKey=FETCH,
>> apiVersion=12, clientId=consumer-mirrormaker2-cluster-1,
>> correlationId=12075) and timeout 30000 to node 1:
>> FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1,
>> maxBytes=52428800, isolationLevel=0, sessionId=1476981493,
>> sessionEpoch=4013, topics=[], forgottenTopicsData=[], rackId='')
>> (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread -
>> mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Received FETCH response from node 0 for request with header
>> RequestHeader(apiKey=FETCH, apiVersion=12,
>> clientId=consumer-mirrormaker2-cluster-1, correlationId=12073):
>> FetchResponseData(throttleTimeMs=0, errorCode=0, sessionId=1395933211,
>> responses=[]) (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Node 0 sent an incremental fetch response with throttleTimeMs = 0 for
>> session 1395933211 with 0 response partition(s), 9 implied partition(s)
>> (org.apache.kafka.clients.FetchSessionHandler) [KafkaBasedLog Work Thread -
>> mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-0 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-12 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-15 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-21 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-18 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-24 at position FetchPosition{offset=0,
>> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-6 at position FetchPosition{offset=1,
>> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-3 at position FetchPosition{offset=2,
>> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Added READ_UNCOMMITTED fetch request for partition
>> mirrormaker2-cluster-offsets-9 at position FetchPosition{offset=1,
>> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
>> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node
>> 35.231.229.1:9094 (id: 0 rack: null)
>> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
>> Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Built incremental fetch (sessionId=1395933211, epoch=4010) for node 0.
>> Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of
>> 9 partition(s) (org.apache.kafka.clients.FetchSessionHandler)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(),
>> implied=(mirrormaker2-cluster-offsets-0, mirrormaker2-cluster-offsets-6,
>> mirrormaker2-cluster-offsets-12, mirrormaker2-cluster-offsets-15,
>> mirrormaker2-cluster-offsets-21, mirrormaker2-cluster-offsets-3,
>> mirrormaker2-cluster-offsets-9, mirrormaker2-cluster-offsets-18,
>> mirrormaker2-cluster-offsets-24)) to broker 35.231.229.1:9094 (id: 0
>> rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
>> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,692 DEBUG [Consumer
>> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
>> Sending FETCH request with header RequestHeader(apiKey=FETCH,
>> apiVersion=12, clientId=consumer-mirrormaker2-cluster-1,
>> correlationId=12076) and timeout 30000 to node 0:
>> FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1,
>> maxBytes=52428800, isolationLevel=0, sessionId=1395933211,
>> sessionEpoch=4010, topics=[], forgottenTopicsData=[], rackId='')
>> (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread -
>> mirrormaker2-cluster-offsets]
>>
>> 2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1,
>> groupId=mirrormaker2-cluster] Received HEARTBEAT response from node
>> 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT,
>> apiVersion=4, clientId=connect-1, correlationId=780):
>> HeartbeatResponseData(throttleTimeMs=0, errorCode=0)
>> (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
>>
>> 2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1,
>> groupId=mirrormaker2-cluster] Received successful Heartbeat response
>> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
>> [DistributedHerder-connect-1-1]
>>
>> 2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Sending metadata request
>> MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
>> name='mirrormaker2-cluster-offsets'),
>> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
>> name='mirrormaker2-cluster-status'),
>> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mmtest'),
>> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
>> name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true,
>> includeClusterAuthorizedOperations=false,
>> includeTopicAuthorizedOperations=false) to node
>> nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack: null)
>> (org.apache.kafka.clients.NetworkClient)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>> 2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5,
>> groupId=null] Sending METADATA request with header
>> RequestHeader(apiKey=METADATA, apiVersion=11, clientId=consumer-null-5,
>> correlationId=17039) and timeout 30000 to node 1:
>> MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
>> name='mirrormaker2-cluster-offsets'),
>> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
>> name='mirrormaker2-cluster-status'),
>> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mmtest'),
>> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
>> name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true,
>> includeClusterAuthorizedOperations=false,
>> includeTopicAuthorizedOperations=false)
>> (org.apache.kafka.clients.NetworkClient)
>> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>> ```
>>
>>
>> Intermittently, I'm seeing -
>> org.apache.kafka.common.errors.DisconnectException
>> when MirrorMaker tries to connect to the broker on the source cluster, even
>> though MM2 is installed on the src connection.
>>
>> Data in the src topic (mmtest on nossl-w, region - us-west1) is not
>> moving to target cluster/topic (mmtest on nossl-e, region - us-east1)
>>
>> Pls note:
>> If i logon to the MirrorMaker pod(in cluster - nossl-w), as expected -
>> I'm able to access the kafka pods on the nossl-w.
>>
>> Any ideas on what needs to be done to debug/resolve this issue & enable
>> KafkaMirrorMaker2 working ?
>>
>> tia!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
