Re: MirrorMaker 2 and Negative Lag

Thanks Ning, I created https://issues.apache.org/jira/browse/KAFKA-12635 . Looks like I don't have permissions to assign it to you though. On Tue, Apr 6, 2021 at 2:13 AM Ning Zhang wrote: > Hello Frank, > > Happy to look into it. Do you mind to open a jira ticket and put your and > Alan's

Re: MirrorMaker 2 and Negative Lag

Hello Frank, Happy to look into it. Do you mind to open a jira ticket and put your and Alan's observations into it, then assign it to me (also would be great to refer this link)? I will take some time to reproduce it when available. Thanks On 2021/03/30 19:08:56, Frank Yi wrote: > Hey

Re: MirrorMaker 2 and Negative Lag

Hey Ning, I believe "if the CG offsets do not contain a pair of , simply sync the offsets from source" could be the problematic behavior here? I'm not very familiar with Mirrormaker's internals, so can't say for sure. As I described previously, this "negative lag" problem happens when a target

Re: MirrorMaker 2 and Negative Lag

Hello Frank, Thanks for helping on analyzing the issue. Regarding when the CG offsets at destination cluster will be updated. From the current codebase, there seems 2 criteria: (1) if the CG offsets do not contain a pair of , simply sync the offsets from source (2) for a pair of , if the

Re: MirrorMaker 2 and Negative Lag

Thanks Alan for the investigation and bug report! On Thu, Mar 25, 2021, 3:25 PM Alan Ning wrote: > Another update on this. I am pretty sure I have found a bug in > MirrorSourceTask. The details are written in > https://issues.apache.org/jira/browse/KAFKA-12558. I hope this helps > others > who

Re: MirrorMaker 2 and Negative Lag

Another update on this. I am pretty sure I have found a bug in MirrorSourceTask. The details are written in https://issues.apache.org/jira/browse/KAFKA-12558. I hope this helps others who have encountered this issue. ... Alan On Thu, Mar 25, 2021 at 9:40 AM Alan Ning wrote: > Just an update

Re: MirrorMaker 2 and Negative Lag

Just an update on this. It has been an adventure. Sam, I think you are right that `consumer.auto.offset.reset:latest` does not work. However, I was also seeing issues with the default behavior (which is consumer.auto.offset.reset:earliest). After a lot of trial and error, I have found that my MM2

Re: MirrorMaker 2 and Negative Lag

Hey Alan, I'm running into the same issue as you and I believe I've figured it out. I noticed that consumer partitions on the destination cluster that have a LOG-END-OFFSET=0 all exhibit this issue. It looks like Mirrormaker's offset sync does not work correctly if the partition is empty.

Re: MirrorMaker 2 and Negative Lag

Hey Alan, I'm running into the same issue as you and I believe I've figured it out. I noticed that consumer partitions on the destination cluster that have a LOG-END-OFFSET=0 all exhibit this issue. It looks like Mirrormaker's offset sync does not work correctly if the partition is empty.

Re: MirrorMaker 2 and Negative Lag

OK. I follow now. Let me try to re-test to see if it makes a difference. Thanks. ... Alan On Wed, Mar 17, 2021 at 5:46 PM Samuel Cantero wrote: > I've found that bug the hard way. FWIW I've migrated several clusters from > kafka 0.10 to kafka 2.x using mm2. So offsets sync work fine for kafka

Re: MirrorMaker 2 and Negative Lag

I've found that bug the hard way. FWIW I've migrated several clusters from kafka 0.10 to kafka 2.x using mm2. So offsets sync work fine for kafka 0.10. Best, On Wed, Mar 17, 2021 at 6:43 PM Samuel Cantero wrote: > No, what I meant is that offsets sync won't work if >

Re: MirrorMaker 2 and Negative Lag

No, what I meant is that offsets sync won't work if `consumer.auto.offset.reset:latest` (it was not talking about that particular bug). Try setting `consumer.auto.offset.reset:earliest` and do verify if offsets are sync'd correctly. Best, On Wed, Mar 17, 2021 at 6:42 PM Alan Ning wrote: > Hey

Re: MirrorMaker 2 and Negative Lag

Hey Ning, My source cluster is a very old cluster running 0.10. I don't think I can change the version. In fact, my whole effort is to migrate out of the legacy cluster to 2.6+. Any insight would be greatly appreciated. Thank you. ... Alan On Wed, Mar 17, 2021 at 4:27 PM Ning Zhang wrote: >

Re: MirrorMaker 2 and Negative Lag

Hey Samuel, I am aware of that `consumer.auto.offset.reset:latest` problem. It was because this PR never made it to trunk. I patched MM2 locally for 2.7 so that `latest` offset will work. ... Alan On Wed, Mar 17, 2021 at 4:50 PM

Re: MirrorMaker 2 and Negative Lag

I've seen this before. I've found that consumer offsets sync does not work with `consumer.auto.offset.reset:latest`. If you set this to earliest, then it should work. One way to workaround the need to start from earliest is by starting with latest and once mirroring is ongoing swap to earliest.

Re: MirrorMaker 2 and Negative Lag

Hello Alan, I may probably see the similar case. One quick validation that could be run is to test on the source cluster with higher Kafka version. If still not working, please email me and I could introduce you to person who may have similar case before. On 2021/03/15 21:59:03, Alan Ning

MirrorMaker 2 and Negative Lag

I am running MirrorMaker 2 (Kafka 2.7), trying to migrate all topics from one cluster to another while preserving through `sync.group.offsets.enabled=true`. My source cluster is running Kafka 0.10, while the target cluster is running 2.6.1. While I can see data being replicated, the data on the