Re: MirrorMaker 2 - Can we start mirroring from latest?

Hey Samuel, I am facing a similar issue with the partitions.assignment.strategy consumer setting on MM2. Let me know if you have any success with setting this (I am also interested with the auto.offset.reset config as well but its a lower priority for my testing. I assume the solution will be

partition.assignment.strategy for MM2?

Hello, Following our experimenting with MM2, we seem to have a hard time recovering from lags created by increased traffic. Since we're running MM2 on kubernetes, we can easily add more resources/instances to our MM2 cluster, which should in theory increase the clusters performance and allow us

MirrorMaker 2 - is starting mirroring from latest possible?

Hello, Whenever we add a new topic to the mirroring whitelist it starts to mirror the entire content of the source topic. If the topic is large this can create a long lag until the entire topic is mirrored, and it can also create some smaller delays on other mirrored topics (i'm assuming this is

MirrorMaker 2 WorkerSourceTask Failed to flush error messages

Hello, I'm seeing large lag sometimes on my MM2 clusters after restarting the cluster (it runs on k8s). I have 3 mm2 clusters, each one reads from 1 source and writes to the same destination. I am seeing these errors on one of my clusters right now. WorkerSourceTask{id=MirrorSourceConnector-33}

Re: Mirrormaker 2 logs - WARN Catching up to assignment's config offset

opics. > > Ryanne > > On Wed, Jul 29, 2020 at 1:03 AM Iftach Ben-Yosef > wrote: > > > Hello > > > > I'm running a mirrormaker 2 cluster which copies from 3 source clusters > > into 1 destination. Yesterday I restarted the cluster and it took 1 of >

Mirrormaker 2 logs - WARN Catching up to assignment's config offset

Hello I'm running a mirrormaker 2 cluster which copies from 3 source clusters into 1 destination. Yesterday I restarted the cluster and it took 1 of the mirrored topics a pretty long time to recover (2~ hours) Since the restart the mm2 cluster has been sending a lot of these warning messages

Re: mirrormaker 2 monitoring

Got it. Thanks! Iftach On Sun, Jul 19, 2020, 19:27 Ryanne Dolan wrote: > Iftach, MM2 uses the same MetricsReporter interface that brokers use, > however seems most people use a JMX exporter. > > Ryanne > > On Sun, Jul 19, 2020, 7:12 AM Iftach Ben-Yosef > wrote: &g

mirrormaker 2 monitoring

Hello, I want to monitor our mirrormaker 2 cluster using prometheus. We are mostly interested in lag, incoming/outgoing traffic and java related metrics. Does anyone have experience with this? Should I use the kafka prometheus jmx exporter or some other exporter for this? (kafka connect exporter

Re: compression on topic vs producer?

ession: > If the cluster names are c1 and c2 then > *c1->c2.producer.override.**compression.type=lz4* > > Thanks, > Nitin > > > On Thu, Jul 9, 2020 at 11:30 AM Iftach Ben-Yosef > wrote: > > > Hello Nitin, I have been unable to successfully setup producer level > &g

Re: compression on topic vs producer?

ion at producer level in mm2 ? Can you > please explain? > > Thanks, > Nitin > > On Thu, Jul 9, 2020 at 10:37 AM Iftach Ben-Yosef > wrote: > > > Hello, > > > > Up until now we have configured compression on producer level. Since > moving > >

compression on topic vs producer?

Hello, Up until now we have configured compression on producer level. Since moving to mm2, we are having some issues with producer level compression and mm2, and are planning on trying out topic level compression for this case. Are there any inherent differences to each method other than

Re: destination topics in mm2 larger than source topic

I believe I got it to work with "source->dest.producer.compression.type = gzip" Is there a way to set this globally for the mm2 process and not to do it per mirroring flow? Thanks, Iftach On Tue, Jul 7, 2020 at 9:34 AM Iftach Ben-Yosef wrote: > Upon further investigati

Re: destination topics in mm2 larger than source topic

k. Thanks, Iftach On Mon, Jul 6, 2020 at 8:03 AM Iftach Ben-Yosef wrote: > Ricardo, > > Thanks for the reply. I did some more testing. I tried mirroring a > different topic from 1 of the 3 source clusters used from the previous > test, into the same destination cluster

Re: destination topics in mm2 larger than source topic

ork failures > that forced the internal producer of MM2 to retry multiple times and hence > produce more data that it should. > > The bottom-line is that certain troubleshooting exercises are hard or > sometimes impossible to diagnose with cases that might have been an outlier. > > --

Re: destination topics in mm2 larger than source topic

nd the mm2 destination topics. Thanks, Iftach On Wed, Jul 1, 2020 at 4:54 PM Ryanne Dolan wrote: > Iftach, is it possible the source topic is compressed? > > Ryanne > > On Wed, Jul 1, 2020, 8:39 AM Iftach Ben-Yosef > wrote: > > > Hello everyone. > > > > I

destination topics in mm2 larger than source topic

Hello everyone. I'm testing mm2 for our cross dc topic replication. We used to do it using mm1 but faced various issues. So far, mm2 is working well, but I have 1 issue which I can't really explain; the destination topic is larger than the source topic. For example, We have 1 topic which on the

Configure Kafka MM2 on Kafka 2.5 to copy from several topics with the same name into 1 destination

Hello, Currently we are using mm1 to copy from several DCs into aggregated kafka clusters. The name of the source and destination topics are the same. MM2 adds the source cluster prefix automatically and this causes issues for our use case. Is there a way to tell MM2 to not create the source