Mirror Maker

Hi All, I am hoping that mirror maker is what I'm looking for. I would like to have complete data center fail over from one kafka cluster to another. If one data center goes down, my producers and consumers will start using the mirrored cluster until the primary is back online. Is that some

mirror-maker

hi, I try to use mirror maker to replicate the data, and I have one trivial question: Do I need access to / add  **all** Kafka Brokers ? I have a 3.1 Cluster with 7 brokers and a new 3.5 with 5 brokers and I want to migrate from the 3.1 (Zoo) to 3.5 (KRaft). Both clusters are in different

KAfka Mirror Maker

Hi, I'm using Mirror Maker with a cluster of 3 nodes and cluster of 5 nodes. I would like to ask - is the number of nodes a restriction for Mirror Maker? Also, are there any other restrictions or properties that should be common across both the clusters so that they continue mirroring.

Re: Mirror Maker

the end of the Kafka topic. The impact is that a small number of messages during the failover window could be lost and the consumer may get a small number of duplicates too. Thanks, Jun On Mon, Sep 23, 2013 at 4:53 PM, Tom Amon wrote: > Hi All, > > I am hoping that mirror maker is

Re: Mirror Maker

duplicates too. > > Thanks, > > Jun > > > On Mon, Sep 23, 2013 at 4:53 PM, Tom Amon wrote: > >> Hi All, >> >> I am hoping that mirror maker is what I'm looking for. I would like to have >> complete data center fail over from one kafka cluster to anoth

SSL Mirror Maker

addresses: PLAINTEXT -> EndPoint(192.168.64.1,9092,PLAINTEXT),SSL -> EndPoint(192.168.64.1,9093,SSL) And I also run below command and can see the proper server's certificate. openssl s_client -debug -connect localhost:9093 -tls1 But When I run mirror maker process on the other mirror

Mirror Maker Errors

Hello, i running mirror maker for mirroring data from one cluster to another. Now i get this error in log Feb 25 22:38:56 ld4-27 MirrorMaker[54827]: [2018-02-25 22:38:56,914] ERROR Error when sending message to topic rc.exchange.jpy with key: 29 bytes, value: 153 bytes with error

Mirror Maker configuration

hi everyone I´m using mirror maker tool and i´m configuring the consumer.properties and producer.properties, but i´m not sure which is the place where i need to put these files. Because mirrormaker tool is installed into the server with kafka manager, and they are two brokers configured as well

Mirror Maker 2.0

Good morning, I would like to have more information about the acquisition of the licence Mirror Maker 2.0 and the commercial support that is offered Thank you very much ! [cid:image001.png@01D87B03.2452BCB0] Josée Brodeur Conseillère Approvisionnement stratégique, Technologies d'aff

Re: mirror-maker

hi, it seems to be pretty clear .. TCPDump says, it tries to connect to the other hosts too, so I need to allow it. cu denny

Re: mirror-maker

gt; I try to use mirror maker to replicate the data, and I have one trivial > question: > > Do I need access to / add **all** Kafka Brokers ? > > I have a 3.1 Cluster with 7 brokers and a new 3.5 with 5 brokers and I > want to migrate from the 3.1 (Zoo) to 3.5 (KRaft). Both cluste

Re: mirror-maker

hi Richard, Am 02.10.23 um 16:46 schrieb Richard Bosch: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Denny, If you use mirrormaker to replicate data between clusters th

questions about mirror maker

Hi, 1. In documentation 0.8.1, the mirror maker command is: bin/kafka-run-class.sh kafka.tools.MirrorMaker --zookeeper zk_host:port/chroot --consumer.config consumer-1.properties --consumer.config consumer-2.properties --producer.config producer.properties --whitelist my-topic

question about mirror maker

Hi, I have a question about mirror maker. say I have 3 data centers each producing topic 'A' with separate kafka cluster running. if 3 of the data need to be kept in sync with each other, shall i create 3 mirror maker in each data center to get the data from the other two ? also, it

OutOfMemoryError in mirror maker

Hi, I am using mirror maker in trunk to replica data across two data centers. While the destination broker was having busy load and unresponsive the send rate of mirror maker was very low and the available producer buffer was quickly filled up. At the end mirror maker threw OOME. Detailed

Re: KAfka Mirror Maker

Mirror Maker does not have specific restrictions on cluster size. The error you saw was because consumer was not able to talk to the broker. Can you try to use kafka-console-consumer to consume some data from your source cluster and see if it works? It should be under KAFKA_HOME/bin/ Jiangjie

Questions on mirror maker

Hi all, I have two questions regarding setting up mirror maker for our cross cluster replication (DC1 to DC2, for instance) 1. In what use case you would want to specify multiplied consumer configs? 2. It seems like the consumer inside the mirror is a SimpleConsumer. Is it possible

Tuning mirror maker performance

I am trying to tune mirrormaker configurations based on this doc and would like know your recommendations. Our configuration: We are doing int

Mirror Maker [Aggregate cluster]

Hi there, I'm sorry about this silly question, just trying to make sure I got this right: Having a separate cluster for aggregate, means that I would need a second zookeeper for that cluster right? I would need a ZK for my local cluster, and then a ZK for my aggregate, is this correct? Regards

Mirror maker SSL support

Hi, folks, I just want to check if anybody know if there are any plan of Kafka MirrorMaker support running over SSL. I tried to follow https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka to configure consumer/producer properties with SSL but no luck, looks that Kafka MirrorMa

Kafka Mirror Maker & Encryption

Hey Guys, I have been hunting through the archives and can't find anyone else that has ran into this issue so I'm going to toss it out there an hope someone can point me to a solution. We are running two kafka clusters on opposite coasts in AWS and need to have mirror maker copy th

Re: SSL Mirror Maker

.64.1,9093,SSL) > > And I also run below command and can see the proper server's certificate. > openssl s_client -debug -connect localhost:9093 -tls1 > > > > But When I run mirror maker process on the other mirror maker machine, > I got error. > > *[Mirror Maker]*

Mirror maker Configs 0.9.0

Hi Experts, I am trying to mirror -- Regards, Prabhu.V

Kafka mirror maker issue

Hi I am doing some experiments on kafka mirror maker. I have two Kafka cluster, one is kafka_2.11-0.9.0.1 with 3 nodes as source cluster, another is kafka_2.10-0.9.0.1 with one node as target cluster. First I mirror the message for topic TEST_TOPIC from source cluster to target cluster with

mirror-maker-thread-failure

Does anyone has experience or had similar issue like https://community.hortonworks.com/questions/106969/mirror-maker-thread-failure-due-to-kafkatoolsmirro.html If you can point me to something that I can try that will be highly appreciated Thank you

Kafka Mirror Maker Automation

Team, Please help me to set it up the Kafka Mirror Maker automation for Kafka Cluster ? Thanks Saravanan

mirror-maker and firewalls

I'm setting up inter-cluster mirroring between cloud and onprem. The zookeepers and brokers are on private subnets. The mirrormaker will run at the target cluster and I'm guessing it needs to talk to both the zookeepers and brokers in the source cluster. I suppose I could set up NAT rules to rea

Mirror Maker 2 problems

Hello, I am trying to setup a replication between 2 clusters with MM2. The goal is to be able to disksnapshot the secondary cluster for backups hence we only want one broker on the secondary cluster. It's very important for us that we can replicate the offsets which MM2 should solve for us. Cluste

[External] Mirror Maker Questions

Hi everyone, I have a few small questions for you regarding MirrorMaker. 1. I know that MirrorMaker 2.0 is only available starting with Kafka version 2.4. Does that mean that if I was mirroring from cluster A to cluster B, that both clusters need to be running Kafka 2.4? 2. For MirrorMaker 1.

Mirror Maker 2 MirrorClient

All: I was tinkering around with the MirrorClient and was curious about the configs. I see that I can use the MirrorMaker config to pull in the same configs that were used to spin up the MM2 cluster. Seeing as MM2 is started with these configs in a property file and it is passed in on command

Kafka Mirror Maker 2

Hey Team, I've a few doubts regarding how the producers work after failover in Mirror Maker 2 Let's say that we have two clusters K1 and K2 and configured MM2 replication for TOPIC1 (originally created in just K1). We configured the active-active replication: K1->K2.enabled

Kafka Mirror Maker 2.0

I was exploring the Mirror maker 2.0. I read through this https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 documentation and I have a few questions. 1) For running mirror maker as a dedicated mirror maker cluster, the documentation specifies a config file and a

Mirror Maker 2.0 Queries

Hi I was exploring the Mirror maker 2.0. I read through this https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 documentation and I have a few questions. 1. For running mirror maker as a dedicated mirror maker cluster, the documentation specifies a config file

RE: Mirror Maker Errors

cks=-1 4. block.on.buffer.full=true Like advised in this topic: https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html I have the following assumption: Your MM can't reach your targeted cluster (rc.exchange.jpy right ?) for this reason, your producer will retry ind

Kafka mirror maker help

Hello, I'm setting up mirror maker for kafka however it doesn't help can you guide me some steps or else share the model shell script to execute the mirror maker ? highly appreciated your help. Thanks Sara

Mirror Maker 2 - Issues

Hi All, I'm trying to use mirror maker 2 to replicate data to our new AWS MSK kafka cluster and I have been running into so many issues and I couldn't find proper documentation. Need some help and it's very urgent. Thanks Also I don't see any of my topics created. Note: Th

Mirror Maker 2 Query

Dear Users, I have a general query regarding MM2. Can MM2 version 3.0 work with Kafka 2.5? Thanks

question: mirror maker 2

Hello all, I am setting up mm2 to replicate messages, consumer groups, and consumer offset from a->b. I believe I am replicating those 3 items from a->b. my mm2 prop file is as followed: ``` # specify any number of cluster aliases clusters = a,b b.group.id=mm2-request # replication settings ta

Re: questions about mirror maker

the value. Guozhang On Mon, Mar 24, 2014 at 2:29 PM, Churu Tang wrote: > Hi, > > 1. In documentation 0.8.1, the mirror maker command is: > bin/kafka-run-class.sh kafka.tools.MirrorMaker --zookeeper > zk_host:port/chroot >--consumer.config consumer-1.properties --

Re: questions about mirror maker

h the defaults, but for some of them > such as the metadata.broker.list and zookeeper.connect string you are > required to specify the value. > > Guozhang > > > On Mon, Mar 24, 2014 at 2:29 PM, Churu Tang >wrote: > > > Hi, > > > > 1. In documentation 0.8.1

Re: question about mirror maker

mirror maker is as fault tolerant as any other consumer. That is, if a mirror maker goes down, the others configured with the same consumer group (we generally run at least 4 for any mirror maker, sometimes up to 10) will rebalance and start back up from the last committed offset. What you need to

Re: question about mirror maker

Hi Todd, Thanks for your answer. with regard to fail over for mirror maker, does that mean if i have 4 mirror maker running in different machines with same consumer group, it will auto load balance if one of the mirror maker fails ? Also, it looks to prevent mirror maker commit wrong (consumer

Re: question about mirror maker

Yes, on both counts. Putting the mirror maker in the same datacenter in the target cluster is exactly what we do as well. We also monitor both the consumer lag (by comparing the offsets stored in Zookeeper and the tail offset on the brokers), and the number of dropped and failed messages on the

Re: question about mirror maker

if placing mirror maker in the same datacenter as target cluster, it/consumer will talks to zookeeper in remote/source datacenter. would it more susceptible to network problems? As for the problem commit offset without actually producing/writing msgs to target cluster, it can be solved by

Re: question about mirror maker

acks setting for the producer). The problem with this is that the consumer and the producer in mirror maker are separated by a queue, and there would need to be a significant amount of communicated added between the two for that to work, but without a loss of throughput. -Todd On 5/12/14, 11:26 AM

Mirror Maker and Client ID

Hi, I'm looking into using mirror maker between two data centers (call them A and B), both with their own Kafka clusters and zookeeper instances. I'm looking into failover and failback scenarios and wondering if Mirror Maker supports any way to not mirror messages from certa

New mirror maker consumer.config question

Hi, I discovered that the new mirror maker implementation in trunk now only accept one consumer.config property instead of a list of them which means we can only supply one source per mirror maker process. Is it a reason for it? If I have multiple source kafka clusters do I need to setup multiple

Usage of Kafka Mirror Maker

irrorMaker process running in one of my datacenter? I see in below link that a better version is proposed https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement Regards, Nitin Kumar Sharma.

Re: OutOfMemoryError in mirror maker

you said, GC should take care of this. In LinkedIn we are using G1GC with some tunings made by our SRE. You can try that if interested. Thanks, Jiangjie (Becket) Qin On 6/13/15, 11:39 AM, "tao xiao" wrote: >Hi, > >I am using mirror maker in trunk to replica data across two d

Re: OutOfMemoryError in mirror maker

You can try that if interested. > > Thanks, > > Jiangjie (Becket) Qin > > On 6/13/15, 11:39 AM, "tao xiao" wrote: > > >Hi, > > > >I am using mirror maker in trunk to replica data across two data centers. > >While the destination broker was havin

Re: OutOfMemoryError in mirror maker

me tunings > > made by our SRE. You can try that if interested. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On 6/13/15, 11:39 AM, "tao xiao" wrote: > > > > >Hi, > > > > > >I am using mirror maker in trunk to r

Re: OutOfMemoryError in mirror maker

As you said, > GC > > > should take care of this. In LinkedIn we are using G1GC with some > tunings > > > made by our SRE. You can try that if interested. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > On 6/13/15

Re: OutOfMemoryError in mirror maker

its size. Can you open a ticket and maybe also >> submit >> > a >> > > patch? >> > > >> > > For unreachable objects I don¹t think it is memory leak. As you >>said, >> GC >> > > should take care of this. In LinkedIn we are

Re: OutOfMemoryError in mirror maker

e probably should fix it as the value is not > >> used > >> > > except logging the its size. Can you open a ticket and maybe also > >> submit > >> > a > >> > > patch? > >> > > > >> > > For unreachable obj

Re: OutOfMemoryError in mirror maker

s, the issue that ErrorLoggingCallback keeps value as local >> >>variable >> >> is >> >> > > known for a while and we probably should fix it as the value is >>not >> >> used >> >> > > except logging the its size. Can you

Re: OutOfMemoryError in mirror maker

gt;> >> > > Hi Tao, > >> >> > > > >> >> > > Yes, the issue that ErrorLoggingCallback keeps value as local > >> >>variable > >> >> is > >> >> > > known for a while and we probably should fix it

Re: OutOfMemoryError in mirror maker

gt;> >> >> > >> >> >> > > Hi Tao, >> >> >> > > >> >> >> > > Yes, the issue that ErrorLoggingCallback keeps value as local >> >> >>variable >> >> >> is >> >

Re: OutOfMemoryError in mirror maker

gt; > Patch submitted > https://issues.apache.org/jira/browse/KAFKA-2281 > >> >> >> > > >> >> >> > On Mon, 15 Jun 2015 at 02:16 Jiangjie Qin > >> >> > >> >> >> > wrote: > >> >> >

Re: OutOfMemoryError in mirror maker

> > > >> >>wrote: > > >> >> >> > > >> >> >> > Thank you for the reply. > > >> >> >> > > > >> >> >> > Patch submitted > > https://issues.apache.org/jira/browse/KAFKA-2281 > > >> &

Re: Questions on mirror maker

Calvin, For (1) can you clarify what you mean by "multiplied" consumer configs? For (2) the mirror-maker actually uses the high level consumer. Thanks, Joel On Sun, Jul 14, 2013 at 7:43 AM, Calvin Lei wrote: > Hi all, > I have two questions regarding setting up mirror mak

Re: Questions on mirror maker

: > Calvin, > > For (1) can you clarify what you mean by "multiplied" consumer configs? > For (2) the mirror-maker actually uses the high level consumer. > > Thanks, > > Joel > > On Sun, Jul 14, 2013 at 7:43 AM, Calvin Lei wrote: > > Hi all, > > I

Re: Tuning mirror maker performance

Hi, I am trying to tune mirrormaker configurations based on this doc and would like know your recommendations. Our configuration: We are doin

Re: Tuning mirror maker performance

You have to determine whether the bottleneck is in the consumer or the producer. To improve the performance of the latter, you can increase the # of total consumer streams. # streams is capped by total # partitions. So, you may need to increase the # of partitions. To improve the performance of t

Fwd: Tuning mirror maker performance

affect picking up any new topics..? Thanks, Raja. -- Forwarded message -- From: Jun Rao Date: Fri, Aug 23, 2013 at 12:08 AM Subject: Re: Tuning mirror maker performance To: "users@kafka.apache.org" You have to determine whether the bottleneck is in the consumer or the pro

Re: Tuning mirror maker performance

ting topic or it affect > picking up any new topics..? > > Thanks, > Raja. > > -- Forwarded message -- > From: Jun Rao > Date: Fri, Aug 23, 2013 at 12:08 AM > Subject: Re: Tuning mirror maker performance > To: "users@kafka.apache.org" > > >

Re: Mirror Maker [Aggregate cluster]

You can also just use a different namespace on the same zookeeper cluster if both local and aggregate are in the same colo. Thanks, Neha On Mon, Nov 18, 2013 at 12:44 PM, Vinicius Carvalho < viniciusccarva...@gmail.com> wrote: > Hi there, I'm sorry about this silly question, just trying to make

Re: Mirror maker SSL support

Currently, MirrorMaker can run the producer with SSL. We will add the option to run the new consumer in MirrorMaker in KAFKA-2452, which will enable SSL on the consumer side too. Thanks, Jun On Fri, Aug 28, 2015 at 12:01 PM, Xiang Zhou (Samuel) wrote: > Hi, folks, > > I just want to check if a

Re: Mirror maker SSL support

Thank you for update Jun. What's the schedule to get new consumer implemented in Mirror Maker? Samuel On Mon, Aug 31, 2015 at 10:06 AM, Jun Rao wrote: > Currently, MirrorMaker can run the producer with SSL. We will add the > option to run the new consumer in MirrorMaker in KAFKA-

Re: Mirror maker SSL support

Probably within 2-3 weeks. Thanks, Jun On Mon, Aug 31, 2015 at 10:54 AM, Xiang Zhou (Samuel) wrote: > Thank you for update Jun. > What's the schedule to get new consumer implemented in Mirror Maker? > > Samuel > > On Mon, Aug 31, 2015 at 10:06 AM, Jun Rao wrote: >

aggregate clusters and mirror maker

hello, i have some questions about setting up an aggregate cluster in a data center for mirror maker. i haven't found answers for these, so i'm appealing to the community at large :-) does the aggregate cluster need its own zookeeper service, or can it use the same service u

mirror maker: mapping topic names

using mirror maker, i would like to write events received from one topic and write them to another. for example, for events received on the topic 'clicks,' i want to write them to 'mirrored.clicks' on my destination cluster. is that possible? thanks, d

Mirror maker stops working (0.8.2.1)

Hello, We have been seeing consistent issue mirroring between our DataCenters. Below is our setup Source: AWS (13 Brokers) Destination: OTHER-DC (20 Brokers) Topic: mirrortest (source: 260 partitions, destination: 200 partitions) Connectivity: AWS Direct Connect (max 6Gb

Re: Mirror maker Configs 0.9.0

Hi Experts, I am trying to replicate data between different data centers using mirror maker tool. kafka-run-class.bat kafka.tools.MirrorMaker --consumer.config consumer.properties --producer.config producer.properties --whitelist * Can someone provide the sample consumer.properties and

Re: Mirror maker Configs 0.9.0

_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname= -Dcom.sun.management.jmxremote.rmi.port=" export JMX_PORT= /usr/bin/kafka-mirror-maker --consumer.c

Re: Mirror maker Configs 0.9.0

ement.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.local.only=false > -Djava.rmi.server.hostname= > -Dcom.sun.management.jmxremote.rmi.port=" > export JMX_PORT= > /usr/bin/kafka-mirror-maker --consumer.config $HOME_DIR/consumer.prop

Re: Mirror maker Configs 0.9.0

anagement.jmxremote=true > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.local.only=false > -Djava.rmi.server.hostname= > -Dcom.sun.management.jmxremote.rmi.port=" > > export JMX_PORT= > >

Re: Mirror maker Configs 0.9.0

I've attached my two configs here. Pay close attention to the --num-streams argument to mirror-maker. I have a lot of thru-put on my topics so I ended up matching the number of streams = number of partitions for each of my topics. A stream is essentially just a consumer and producer thread

RE: Mirror maker Configs 0.9.0

Thanks a lot Stephen.. It worked:) -Original Message- From: Stephen Powis [mailto:spo...@salesforce.com] Sent: Wednesday, March 09, 2016 6:58 PM To: users@kafka.apache.org Cc: Kishore N R Subject: Re: Mirror maker Configs 0.9.0 I've attached my two configs here. Pay close atte

Lag in Kafka Mirror-maker

Hi, We are using kakfa mirror-maker to replicate kakfa between two data centers. Kafka have 15 topics with 5 partitions each. Mirror-maker have 5 producers and 5 streams and queue.enqueue.timeout.ms=-1 and its running on target data center. The normal lag offsets between source and target kafka is

Re: Mirror maker Configs 0.9.0

Hello Experts, I am trying to replicate data between On Prem Kafka Cluster(Source) and another Kafka cluster (Target?) set up in a Cloud provider environment. The On-Prem to cloud is connected via IPSec VPN and Mirror maker tool is used on 0.10.2.x version. The consumer configs are as

New consumer - mirror maker - blacklist

Hi All, The docs currently say the following: However, --blacklist is not supported when the new consumer has been enabled (i.e. when bootstrap.servers has been defined in the consumer configuration) Is there an alternative to using a blacklist when using the new consumer type? So that it is pos

Re: Kafka Mirror Maker Automation

Have yu got a chance to help to me on this issue ? Thanks Saravanan From: "Kannappan, Saravanan (Contractor)" Date: Thursday, September 21, 2017 at 7:11 PM To: "users@kafka.apache.org" Subject: Kafka Mirror Maker Automation Team, Please help me to set it up the Kafka Mir

Re: Kafka Mirror Maker Automation

(Contractor) < saravanan_kannap...@comcast.com> wrote: > Have yu got a chance to help to me on this issue ? > > Thanks > Saravanan > > From: "Kannappan, Saravanan (Contractor)" > > Date: Thursday, September 21, 2017 at 7:11 PM > To: "users@kafka.apach

Mirror maker setup - multi node

Hi All, So I understand I can run the following to aggregate topics from two different clusters into a mirror: bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClu

Preserve offset in Mirror Maker

We're using mirror maker to mirror data from one data center to another data center ( 1 to 1 ). We noticed that by default the mirror maker by default start from latest offset. How to change mirror maker producer config to start from last check pointed offset in case of crash without l

Kafka Mirror maker duplicate issue

Hi there, I created a broker as stand by using Kafka Mirror maker but same messages gets consumed by both Source broker and mirror broker. Ex: I send 1000 messages let's say offset value 1 to 1000 and consumed 500 messages from the source broker. Now my broker goes down and want to read

Mirror Maker - Message Format Issue?

seeing errors from the mirror maker when trying to produce to the destination cluster like the following: java.lang.IllegalArgumentException: Invalid timestamp -1 at org.apache.kafka.clients.producer.ProducerRecord.(ProducerRecord.java:60) Is the root problem the 0.8.2 producer sending dat

Re: Mirror Maker 2 problems

Hi Nils Can you try with the latest mm2 default config file found at https://github.com/apache/kafka/blob/trunk/config/connect-mirror-maker.properties. Please feel free to reach out again if you are still stuck. Thanks Karan On Tue, Jan 14, 2020 at 1:48 PM Nils Hermansson < nils.hermans...@get

Re: Mirror Maker 2 problems

Hello, yes that seems todo the trick. Another questions is there any documentation somewhere describing how the offset translations topics MM2 creates actually works? We need to write code ourself until this PR is merged I guess. https://github.com/apache/kafka/pull/7577 Thanks for the help. On

Re: Mirror Maker 2 problems

Another question is there anyway to open up a rest port like standard 8083 when running ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties have tried adding rest.port=8083 but does not seem to work. Would like to be able to query the status of the connector via rest. On Tue, J

Re: Mirror Maker 2 problems

Nils, a REST API for MM2 was discussed at length, but so far the community hasn't settled on a plan. For now, if you need to monitor MM2's Connectors over HTTP, you can use a bunch of Connect clusters and manually configure them to run MirrorSourceConnector etc. This has some advantages, e.g. you c

Re: [External] Mirror Maker Questions

On #2, you can provide an implementation of a MirrorMakerMessageHandler that will called for each record - you ensure it's in the classpath and pass the class name to MM using --message.handler. On Thu, 20 Feb. 2020, 9:49 pm Dean Barlan, wrote: > Hi everyone, > > I have a few small questions f

Re: [External] Mirror Maker Questions

For 1), MM2 will work with older versions of Kafka. I've gotten it to work with clusters as old as 0.10.2 but with some features disabled iirc. Ryanne On Thu, Feb 20, 2020, 2:49 AM Dean Barlan wrote: > Hi everyone, > > I have a few small questions for you regarding MirrorMaker. > > 1. I know t

Mirror Maker 2 internal topics

Hello, I am prototyping Kafka replication with Mirror Maker 2. At the beginning, I had hard times with org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'mm2-offsets.dst.internal'... Caused by: java.util.concurrent.Executio

Re: Mirror Maker 2 MirrorClient

Hey Carl, that's what my team has done for our internal tooling, and I designed MirrorClient with that in mind. Given a single mm2.properties file you can create MirrorClients for each cluster and those in turn give you Admin/Consumer/Producer clients if you need them. Our internal tooling essentia

Re: Kafka Mirror Maker 2

ding how the producers work after failover in Mirror > Maker 2 > > Let's say that we have two clusters K1 and K2 and configured MM2 > replication for TOPIC1 (originally created in just K1). > > We configured the active-active replication: > > K1->K2.enabled = tru

Re: Kafka Mirror Maker 2

Hi Himanshu Can u pls tell how to use MM2.. I am using Apache Kafka,in this normal mirror maker is only available.. Most of the people saying like to use MM2 but I didn't able to know where to get that MM2. Is it related to Apache or from some other Distributors? Can u pls explain h

Re: Mirror Maker 2.0 Queries

Ananya, yes the driver is distributed, but each worker only communicates via kafka. They do not listen on any ports. Ryanne On Sat, Jul 11, 2020, 11:28 AM Ananya Sen wrote: > Hi > > I was exploring the Mirror maker 2.0. I read through this > > https://cwiki.apache.org/confluenc

Re: Mirror Maker 2.0 Queries

Thank you Ryanne for the quick response. I further want to clarify a few points. The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka connect we have multiple workers and each worker has some assigned task. To map this to Mirror Maker 2.0, A mirror Maker will driver have some

Re: Mirror Maker 2.0 Queries

Any help here would be greatly appreciated. On Sat, Aug 8, 2020, 12:13 PM Ananya Sen wrote: > Thank you Ryanne for the quick response. > I further want to clarify a few points. > > The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka > connect we have multiple w

  1   2   3   4   >