Re: MirrorMaker2 on GKE

a-apps/blob/master/strimzi/kafka/ephemeral-single-with-external-listener.yaml This should be deployed on TARGET cluster. > https://github.com/yuwtennis/apache-kafka-apps/blob/master/strimzi/mm2/plain-source-tls-target.yaml Thanks, Yu On Sun, Feb 26, 2023 at 10:29 PM Yu Watanabe wrote: > >

Re: MirrorMaker2 on GKE

2023-02-23 22:05:54,692 DEBUG [Consumer > >> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster] > >> Built incremental fetch (sessionId=1395933211, epoch=4010) for node 0. > >> Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of > >> 9 partition(s) (org.apache.kafka.clients.FetchSessionHandler) > >> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets] > >> > >> 2023-02-23 22:05:54,692 DEBUG [Consumer > >> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster] > >> Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(), > >> implied=(mirrormaker2-cluster-offsets-0, mirrormaker2-cluster-offsets-6, > >> mirrormaker2-cluster-offsets-12, mirrormaker2-cluster-offsets-15, > >> mirrormaker2-cluster-offsets-21, mirrormaker2-cluster-offsets-3, > >> mirrormaker2-cluster-offsets-9, mirrormaker2-cluster-offsets-18, > >> mirrormaker2-cluster-offsets-24)) to broker 35.231.229.1:9094 (id: 0 > >> rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher) > >> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets] > >> > >> 2023-02-23 22:05:54,692 DEBUG [Consumer > >> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster] > >> Sending FETCH request with header RequestHeader(apiKey=FETCH, > >> apiVersion=12, clientId=consumer-mirrormaker2-cluster-1, > >> correlationId=12076) and timeout 3 to node 0: > >> FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1, > >> maxBytes=52428800, isolationLevel=0, sessionId=1395933211, > >> sessionEpoch=4010, topics=[], forgottenTopicsData=[], rackId='') > >> (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread - > >> mirrormaker2-cluster-offsets] > >> > >> 2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1, > >> groupId=mirrormaker2-cluster] Received HEARTBEAT response from node > >> 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, > >> apiVersion=4, clientId=connect-1, correlationId=780): > >> HeartbeatResponseData(throttleTimeMs=0, errorCode=0) > >> (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1] > >> > >> 2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1, > >> groupId=mirrormaker2-cluster] Received successful Heartbeat response > >> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) > >> [DistributedHerder-connect-1-1] > >> > >> 2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5, > >> groupId=null] Sending metadata request > >> MetadataRequestData(topics=[MetadataRequestTopic(topicId=AA, > >> name='mirrormaker2-cluster-offsets'), > >> MetadataRequestTopic(topicId=AA, > >> name='mirrormaker2-cluster-status'), > >> MetadataRequestTopic(topicId=AA, name='mmtest'), > >> MetadataRequestTopic(topicId=AA, > >> name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true, > >> includeClusterAuthorizedOperations=false, > >> includeTopicAuthorizedOperations=false) to node > >> nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack: null) > >> (org.apache.kafka.clients.NetworkClient) > >> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0] > >> 2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5, > >> groupId=null] Sending METADATA request with header > >> RequestHeader(apiKey=METADATA, apiVersion=11, clientId=consumer-null-5, > >> correlationId=17039) and timeout 3 to node 1: > >> MetadataRequestData(topics=[MetadataRequestTopic(topicId=AA, > >> name='mirrormaker2-cluster-offsets'), > >> MetadataRequestTopic(topicId=AA, > >> name='mirrormaker2-cluster-status'), > >> MetadataRequestTopic(topicId=AA, name='mmtest'), > >> MetadataRequestTopic(topicId=AA, > >> name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true, > >> includeClusterAuthorizedOperations=false, > >> includeTopicAuthorizedOperations=false) > >> (org.apache.kafka.clients.NetworkClient) > >> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0] > >> ``` > >> > >> > >> Intermittently, I'm seeing - > >> org.apache.kafka.common.errors.DisconnectException > >> when MirrorMaker tries to connect to the broker on the source cluster, even > >> though MM2 is installed on the src connection. > >> > >> Data in the src topic (mmtest on nossl-w, region - us-west1) is not > >> moving to target cluster/topic (mmtest on nossl-e, region - us-east1) > >> > >> Pls note: > >> If i logon to the MirrorMaker pod(in cluster - nossl-w), as expected - > >> I'm able to access the kafka pods on the nossl-w. > >> > >> Any ideas on what needs to be done to debug/resolve this issue & enable > >> KafkaMirrorMaker2 working ? > >> > >> tia! > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > > -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe1/ twitter: twitter.com/yuwtennis

Re: New To Kafka - How to Start

Hello . How about starting with 'quick start' ? https://kafka.apache.org/quickstart Thanks, Yu Watanabe On Tue, Aug 10, 2021 at 10:44 AM Gilbert Flores < gilbert.flo...@primergrp.com> wrote: > Hi, > > Good day to you. We are on the stage of exploring integration syste

Re: How to set consumer.overrides.auto.offset.reset as latest in MM2 ?

~/Development/kafka_2.13-2.4.0$ bin/kafka-console-consumer.sh --bootstrap-server migration-kafka-topics-. servicebus.windows.net:9093 --topic 001_status_info --consumer.config consumer.properties --from-beginning ^CProcessed a total of 0 messages ``` Thanks, Yu Watanabe On Tue, Oct 13,

How to set consumer.overrides.auto.offset.reset as latest in MM2 ?

However, I'd really need to use this option. Would there be any way to set 'consumer.overrides.auto.offset.reset' to 'latest ' in kafka 2.4.0 , perhaps by replacing the jar file ? Best Regards, Yu Watanabe -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe

Re: AdminClient fails to authenticate to Azure Eventhub for MirrorMaker2

This seems to be related to the strimzi side. I will continue investigating with stimzi . Thanks. On Sun, Sep 27, 2020 at 3:00 PM Yu Watanabe wrote: > Hello . > > I use MirrorMaker2 on strimzi to mirror data from Azure Eventhub to kafka > cluster. > However, MirrorMaker2 fails

AdminClient fails to authenticate to Azure Eventhub for MirrorMaker2

error but bit having difficulty. Would there be anyone who had faced similar issue ? If so , I appreciate if I could get advice to overcome the error. Best Regards, Yu Watanabe -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe1/ twitter: twitter.com/yuwtennis

How does kafka broker treat log.message.format.version if -lVx is not specified?

ight place... https://github.com/apache/kafka/blob/7f90a58b69cd0eb63ba122b41e6ef6195b0a5d98/core/src/main/scala/kafka/server/KafkaConfig.scala#L126 Thanks, Yu -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe1/ twitter: twitter.com/yuwtennis

Re: How to manually start ingesting in kafka source connector ?

he connector elsewhere to see if you can pre-empt > the key value that Kafka Connect will use when writing the offsets, and so > do your list 2 - 1 - 3 instead > > > -- > > Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff > > > On Thu, 28 May 2020

Re: How to manually start ingesting in kafka source connector ?

veloper Advocate | ro...@confluent.io | @rmoff > > > On Thu, 28 May 2020 at 04:12, Yu Watanabe wrote: > > > Dear community . > > > > I would like to ask question related to source connector in kafka > > connect (2.4.0) . > > > > Is there a way to manually start sour

How to manually start ingesting in kafka source connector ?

/restapi.html#put--connectors-(string-name)-pause however, could not find set initial state for individual tasks in connector properties .. https://docs.confluent.io/current/connect/managing/configuring.html I appreciate if I could get some help. Best Regards, Yu Watanabe -- Yu Watanabe linkedin

Re: When does kafka connect commit messages to kafka broker ?

I appreciate if I could get some advice with this if anyone had experienced similar issue. Thanks, Yu Watanabe On Wed, Jan 15, 2020 at 10:53 PM Yu Watanabe wrote: > > I found the reason. I am not sure if this is Azure problem or Jdbc > problem though... > > First of all my apo

Re: When does kafka connect commit messages to kafka broker ?

ect.jdbc.JdbcSourceConnector", "connection.url" : "jdbc:postgresql://:5432/YYY?user=&password=&ssl=true&readOnly=true&defaultRowFetchSize=5000", "incrementing.column.name" : "sid", "validate.non.null": false, "query": "SELECT sid, m FRO

When does kafka connect commit messages to kafka broker ?

broker using strimzi (0.15.0 , kafka 2.3.1 ). When does kafka connect create topic to store messages in broker ? Thanks, Yu Watnaabe -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe1/ twitter: twitter.com/yuwtennis

Re: Which container should you use when deploying on docker ?

wasn't one so I asked the question. Thanks, Yu Watanabe On Tue, Dec 17, 2019 at 12:26 AM Robin Moffatt wrote: > There are various Kafka images available, including: > > https://hub.docker.com/r/confluentinc/cp-kafka/ > <https://hub.docker.com/r/wurstmeister/kafka/> > h

Replacing custom value while masking in Kafka connect

+replacement+for+MaskField+SMT [WIP] KAFKA-6755: Allow literal value for MaskField SMT #6284 https://github.com/apache/kafka/pull/6284 Best Regards, Yu Watanabe -- Yu Watanabe Weekend Freelancer who loves to challenge building data platform yu.w.ten...@gmail.com [image: LinkedIn icon] <ht

Re: Missing link in online document

rmation, see Parallelism Model. On Mon, Dec 16, 2019 at 3:16 AM Guozhang Wang wrote: > Hello Yu, > > Could you point to me which page has the reference to this link? > > > Guozhang > > On Sun, Dec 15, 2019 at 2:24 AM Yu Watanabe wrote: > > > Hello. > > > > I w

Missing link in online document

t link for kafka stream example ? Thanks, Yu Watanabe -- Yu Watanabe Weekend Freelancer who loves to challenge building data platform yu.w.ten...@gmail.com [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image: Twitter icon] <https://twitter.com/yuwtennis>

Which container should you use when deploying on docker ?

Repository "confluent" seems be the closest one since its the creator of kafka but it does not have above tag . Thanks, Yu Watanabe -- Yu Watanabe Weekend Freelancer who loves to challenge building data platform yu.w.ten...@gmail.com [image: LinkedIn icon] <https://www.linkedin.com/i

Re: How frequent does the follower replica issue a fetch request to leader replica?

Hello Peter. Thank you for the reply. Now I understand. I take a look at the " kafka.server,type=replica-fetcher-metrics" as well. Thanks, Yu Watanabe On Wed, Apr 11, 2018 at 4:48 PM, Peter Bukowinski wrote: > Hi Yu, > > The broker property ‘replica.fetch.wait.max.ms’ det

How frequent does the follower replica issue a fetch request to leader replica?

ncy of the fetch request. How often does follower replica issues fetch request to leader ? Thanks, Yu -- Yu Watanabe 渡辺 裕 LinkedIn : jp.linkedin.com/in/yuwatanabe1