Re: Mirror Maker 2 - offset sync from source to target

[Greg Harris]

Alexander,

Looking at the MM2 documentation
https://kafka.apache.org/documentation/#georeplication it appears that
there's no "summary" of the auto-sync functionality, only the
documentation strings for the individual configurations.
Point (5) above is not covered in the public documentation, and you
would need to read the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
to learn about it.
Point (6) above is also not covered in the public documentation, but
has always been a hidden limitation of MirrorMaker2 for a long time
that has recently become more apparent. We still have some work to do
in order to make MirrorMaker2 more resilient to restarts.

Thanks!
Greg

On Mon, Oct 23, 2023 at 9:00 PM Alexander Shapiro (ashapiro)
 wrote:
>
> Not a problem Greg.
>
> Is there some documentation explaining the same ?
> I tried to find a lot in the past
>
> 
> From: Greg Harris 
> Sent: Monday, October 23, 2023 11:23:26 PM
> To: users@kafka.apache.org 
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> [You don't often get email from greg.har...@aiven.io.invalid. Learn why this 
> is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> CAUTION: This email is from an external source. Please don’t open any unknown 
> links or attachments.
>
> Alexander,
>
> My apologies for calling you Andrew.
>
> Greg
>
> On Mon, Oct 23, 2023 at 1:22 PM Greg Harris  wrote:
> >
> > Andrew,
> >
> > Yes, there isn't an explicit "create consumer group" operation, it
> > should be created when MM2 emits a sync for it.
> >
> > Best,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
> >  wrote:
> > >
> > > Thanks, one clarification plz
> > >
> > > In bullet for You mention "4. The target group does not exist, or has no 
> > > active consumers"
> > > If group on target does not exist, will it be created without active 
> > > consumers ?
> > >
> > > -Original Message-
> > > From: Greg Harris 
> > > Sent: Monday, October 23, 2023 8:56 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > [You don't often get email from greg.har...@aiven.io.invalid. Learn why 
> > > this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > CAUTION: This email is from an external source. Please don’t open any 
> > > unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Sorry I noticed an inconsistency in my last email. For point 6:
> > > If the MirrorCheckpointTask restarts after replication but before offset 
> > > is translated, then it may not be able to perform a translation.
> > > If the MirrorCheckpointTask does not restart, it should be able to 
> > > perform translation.
> > >
> > > So if your MirrorMaker2 is restarting frequently, that may prevent 
> > > consistent translation.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) 
> > >  wrote:
> > > >
> > > > Hi Greg,
> > > > Thank you very much,
> > > > it is the most detailed answer I would expect.
> > > >
> > > > -Original Message-
> > > > From: Greg Harris 
> > > > Sent: Monday, October 23, 2023 8:42 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > > >
> > > > CAUTION: This email is from an external source. Please don’t open any 
> > > > unknown links or attachments.
> > > >
> > > > Hi Alexander,
> > > >
> > > > Thanks for using MirrorMaker2!
> > > >
> > > > If you turn on `sync.group.offsets.enabled`, then the
> > > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > > true:
> > > > 1. The source group exists
> > > > 2. The source group name matches the configured group filter 
> > > > (group.filter.class, groups, groups.exclude) 3. The source group has an 
> > > > offset for a topic which matches the configured topic filter 
> > > > (topic.filter.class, topics, topics.exclude) 4. The target group does 
> > > > not exist, or has no active consu

Re: Mirror Maker 2 - offset sync from source to target

[Alexander Shapiro (ashapiro)]

Not a problem Greg.

Is there some documentation explaining the same ?
I tried to find a lot in the past


From: Greg Harris 
Sent: Monday, October 23, 2023 11:23:26 PM
To: users@kafka.apache.org 
Subject: Re: Mirror Maker 2 - offset sync from source to target

[You don't often get email from greg.har...@aiven.io.invalid. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email is from an external source. Please don’t open any unknown 
links or attachments.

Alexander,

My apologies for calling you Andrew.

Greg

On Mon, Oct 23, 2023 at 1:22 PM Greg Harris  wrote:
>
> Andrew,
>
> Yes, there isn't an explicit "create consumer group" operation, it
> should be created when MM2 emits a sync for it.
>
> Best,
> Greg
>
> On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
>  wrote:
> >
> > Thanks, one clarification plz
> >
> > In bullet for You mention "4. The target group does not exist, or has no 
> > active consumers"
> > If group on target does not exist, will it be created without active 
> > consumers ?
> >
> > -Original Message-----
> > From: Greg Harris 
> > Sent: Monday, October 23, 2023 8:56 PM
> > To: users@kafka.apache.org
> > Subject: Re: Mirror Maker 2 - offset sync from source to target
> >
> > [You don't often get email from greg.har...@aiven.io.invalid. Learn why 
> > this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > CAUTION: This email is from an external source. Please don’t open any 
> > unknown links or attachments.
> >
> > Hi Alexander,
> >
> > Sorry I noticed an inconsistency in my last email. For point 6:
> > If the MirrorCheckpointTask restarts after replication but before offset is 
> > translated, then it may not be able to perform a translation.
> > If the MirrorCheckpointTask does not restart, it should be able to perform 
> > translation.
> >
> > So if your MirrorMaker2 is restarting frequently, that may prevent 
> > consistent translation.
> >
> > Thanks,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) 
> >  wrote:
> > >
> > > Hi Greg,
> > > Thank you very much,
> > > it is the most detailed answer I would expect.
> > >
> > > -Original Message-
> > > From: Greg Harris 
> > > Sent: Monday, October 23, 2023 8:42 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > CAUTION: This email is from an external source. Please don’t open any 
> > > unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Thanks for using MirrorMaker2!
> > >
> > > If you turn on `sync.group.offsets.enabled`, then the
> > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > true:
> > > 1. The source group exists
> > > 2. The source group name matches the configured group filter 
> > > (group.filter.class, groups, groups.exclude) 3. The source group has an 
> > > offset for a topic which matches the configured topic filter 
> > > (topic.filter.class, topics, topics.exclude) 4. The target group does not 
> > > exist, or has no active consumers 5. The target group has no offset for a 
> > > specified partition, or the offset is earlier than the translated offset 
> > > 6. MirrorCheckpointTask restarted after replication happened but before 
> > > the offset could be translated.
> > >
> > > If one of these isn't true, you won't see translation happening.
> > >
> > > Are you having a problem with too many consumer groups being created?
> > > You can restrict the group or topic filters, as they're very permissive 
> > > by default.
> > > Or is the problem that you expect the consumer groups to be created, but 
> > > they aren't? One of the above conditions may not be satisfied, or your 
> > > MirrorCheckpointConnector/Task may be malfunctioning and you'll need to 
> > > inspect the logs to debug it.
> > >
> > > Best,
> > > Greg Harris
> > >
> > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) 
> > >  wrote:
> > > >
> > > > Hi
> > > > Can someone advise please
> > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > target for particular consumer group That 

Re: Mirror Maker 2 - offset sync from source to target

[sherry]

Offset is an excellent artist

On Mon, Oct 23, 2566 BE at 22:24 Greg Harris 
wrote:

> Alexander,
>
> My apologies for calling you Andrew.
>
> Greg
>
> On Mon, Oct 23, 2023 at 1:22 PM Greg Harris  wrote:
> >
> > Andrew,
> >
> > Yes, there isn't an explicit "create consumer group" operation, it
> > should be created when MM2 emits a sync for it.
> >
> > Best,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
> >  wrote:
> > >
> > > Thanks, one clarification plz
> > >
> > > In bullet for You mention "4. The target group does not exist, or has
> no active consumers"
> > > If group on target does not exist, will it be created without active
> consumers ?
> > >
> > > -----Original Message-
> > > From: Greg Harris 
> > > Sent: Monday, October 23, 2023 8:56 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > [You don't often get email from greg.har...@aiven.io.invalid. Learn
> why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > >
> > > CAUTION: This email is from an external source. Please don’t open any
> unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Sorry I noticed an inconsistency in my last email. For point 6:
> > > If the MirrorCheckpointTask restarts after replication but before
> offset is translated, then it may not be able to perform a translation.
> > > If the MirrorCheckpointTask does not restart, it should be able to
> perform translation.
> > >
> > > So if your MirrorMaker2 is restarting frequently, that may prevent
> consistent translation.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) <
> alexander.shap...@amdocs.com.invalid> wrote:
> > > >
> > > > Hi Greg,
> > > > Thank you very much,
> > > > it is the most detailed answer I would expect.
> > > >
> > > > -Original Message-
> > > > From: Greg Harris 
> > > > Sent: Monday, October 23, 2023 8:42 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > > >
> > > > CAUTION: This email is from an external source. Please don’t open
> any unknown links or attachments.
> > > >
> > > > Hi Alexander,
> > > >
> > > > Thanks for using MirrorMaker2!
> > > >
> > > > If you turn on `sync.group.offsets.enabled`, then the
> > > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > > true:
> > > > 1. The source group exists
> > > > 2. The source group name matches the configured group filter
> (group.filter.class, groups, groups.exclude) 3. The source group has an
> offset for a topic which matches the configured topic filter
> (topic.filter.class, topics, topics.exclude) 4. The target group does not
> exist, or has no active consumers 5. The target group has no offset for a
> specified partition, or the offset is earlier than the translated offset 6.
> MirrorCheckpointTask restarted after replication happened but before the
> offset could be translated.
> > > >
> > > > If one of these isn't true, you won't see translation happening.
> > > >
> > > > Are you having a problem with too many consumer groups being created?
> > > > You can restrict the group or topic filters, as they're very
> permissive by default.
> > > > Or is the problem that you expect the consumer groups to be created,
> but they aren't? One of the above conditions may not be satisfied, or your
> MirrorCheckpointConnector/Task may be malfunctioning and you'll need to
> inspect the logs to debug it.
> > > >
> > > > Best,
> > > > Greg Harris
> > > >
> > > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) <
> alexander.shap...@amdocs.com.invalid> wrote:
> > > > >
> > > > > Hi
> > > > > Can someone advise please
> > > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > > target for particular consumer group That group must be created on
> target, even if no actual consumption will be done ?
> > > > >
> > > > >
> > > > >
> > > > > This email and the information contained herein is proprietary and
> > > > > confidential and subject to the Amdocs Email Terms of Service,
> which
> > > > > you may review at
> > > > > https://www.amdocs.com/about/email-terms-of-service
> > > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > > This email and the information contained herein is proprietary and
> > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > you may review at
> https://www.amdocs.com/about/email-terms-of-service
> > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which you
> may review at https://www.amdocs.com/about/email-terms-of-service <
> https://www.amdocs.com/about/email-terms-of-service>
>

Re: Mirror Maker 2 - offset sync from source to target

[Greg Harris]

Alexander,

My apologies for calling you Andrew.

Greg

On Mon, Oct 23, 2023 at 1:22 PM Greg Harris  wrote:
>
> Andrew,
>
> Yes, there isn't an explicit "create consumer group" operation, it
> should be created when MM2 emits a sync for it.
>
> Best,
> Greg
>
> On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
>  wrote:
> >
> > Thanks, one clarification plz
> >
> > In bullet for You mention "4. The target group does not exist, or has no 
> > active consumers"
> > If group on target does not exist, will it be created without active 
> > consumers ?
> >
> > -Original Message-
> > From: Greg Harris 
> > Sent: Monday, October 23, 2023 8:56 PM
> > To: users@kafka.apache.org
> > Subject: Re: Mirror Maker 2 - offset sync from source to target
> >
> > [You don't often get email from greg.har...@aiven.io.invalid. Learn why 
> > this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > CAUTION: This email is from an external source. Please don’t open any 
> > unknown links or attachments.
> >
> > Hi Alexander,
> >
> > Sorry I noticed an inconsistency in my last email. For point 6:
> > If the MirrorCheckpointTask restarts after replication but before offset is 
> > translated, then it may not be able to perform a translation.
> > If the MirrorCheckpointTask does not restart, it should be able to perform 
> > translation.
> >
> > So if your MirrorMaker2 is restarting frequently, that may prevent 
> > consistent translation.
> >
> > Thanks,
> > Greg
> >
> > On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) 
> >  wrote:
> > >
> > > Hi Greg,
> > > Thank you very much,
> > > it is the most detailed answer I would expect.
> > >
> > > -Original Message-
> > > From: Greg Harris 
> > > Sent: Monday, October 23, 2023 8:42 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror Maker 2 - offset sync from source to target
> > >
> > > CAUTION: This email is from an external source. Please don’t open any 
> > > unknown links or attachments.
> > >
> > > Hi Alexander,
> > >
> > > Thanks for using MirrorMaker2!
> > >
> > > If you turn on `sync.group.offsets.enabled`, then the
> > > MirrorCheckpointTask will sync the offsets if all of the following is
> > > true:
> > > 1. The source group exists
> > > 2. The source group name matches the configured group filter 
> > > (group.filter.class, groups, groups.exclude) 3. The source group has an 
> > > offset for a topic which matches the configured topic filter 
> > > (topic.filter.class, topics, topics.exclude) 4. The target group does not 
> > > exist, or has no active consumers 5. The target group has no offset for a 
> > > specified partition, or the offset is earlier than the translated offset 
> > > 6. MirrorCheckpointTask restarted after replication happened but before 
> > > the offset could be translated.
> > >
> > > If one of these isn't true, you won't see translation happening.
> > >
> > > Are you having a problem with too many consumer groups being created?
> > > You can restrict the group or topic filters, as they're very permissive 
> > > by default.
> > > Or is the problem that you expect the consumer groups to be created, but 
> > > they aren't? One of the above conditions may not be satisfied, or your 
> > > MirrorCheckpointConnector/Task may be malfunctioning and you'll need to 
> > > inspect the logs to debug it.
> > >
> > > Best,
> > > Greg Harris
> > >
> > > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) 
> > >  wrote:
> > > >
> > > > Hi
> > > > Can someone advise please
> > > > if sync.group.offsets.enabled : true to sync offset from source to
> > > > target for particular consumer group That group must be created on 
> > > > target, even if no actual consumption will be done ?
> > > >
> > > >
> > > >
> > > > This email and the information contained herein is proprietary and
> > > > confidential and subject to the Amdocs Email Terms of Service, which
> > > > you may review at
> > > > https://www.amdocs.com/about/email-terms-of-service
> > > > <https://www.amdocs.com/about/email-terms-of-service>
> > > This email and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs Email Terms of Service, which
> > > you may review at https://www.amdocs.com/about/email-terms-of-service
> > > <https://www.amdocs.com/about/email-terms-of-service>
> > This email and the information contained herein is proprietary and 
> > confidential and subject to the Amdocs Email Terms of Service, which you 
> > may review at https://www.amdocs.com/about/email-terms-of-service 
> > <https://www.amdocs.com/about/email-terms-of-service>

Re: Mirror Maker 2 - offset sync from source to target

[Greg Harris]

Andrew,

Yes, there isn't an explicit "create consumer group" operation, it
should be created when MM2 emits a sync for it.

Best,
Greg

On Mon, Oct 23, 2023 at 1:15 PM Alexander Shapiro (ashapiro)
 wrote:
>
> Thanks, one clarification plz
>
> In bullet for You mention "4. The target group does not exist, or has no 
> active consumers"
> If group on target does not exist, will it be created without active 
> consumers ?
>
> -Original Message-
> From: Greg Harris 
> Sent: Monday, October 23, 2023 8:56 PM
> To: users@kafka.apache.org
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> [You don't often get email from greg.har...@aiven.io.invalid. Learn why this 
> is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> CAUTION: This email is from an external source. Please don’t open any unknown 
> links or attachments.
>
> Hi Alexander,
>
> Sorry I noticed an inconsistency in my last email. For point 6:
> If the MirrorCheckpointTask restarts after replication but before offset is 
> translated, then it may not be able to perform a translation.
> If the MirrorCheckpointTask does not restart, it should be able to perform 
> translation.
>
> So if your MirrorMaker2 is restarting frequently, that may prevent consistent 
> translation.
>
> Thanks,
> Greg
>
> On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) 
>  wrote:
> >
> > Hi Greg,
> > Thank you very much,
> > it is the most detailed answer I would expect.
> >
> > -Original Message-
> > From: Greg Harris 
> > Sent: Monday, October 23, 2023 8:42 PM
> > To: users@kafka.apache.org
> > Subject: Re: Mirror Maker 2 - offset sync from source to target
> >
> > CAUTION: This email is from an external source. Please don’t open any 
> > unknown links or attachments.
> >
> > Hi Alexander,
> >
> > Thanks for using MirrorMaker2!
> >
> > If you turn on `sync.group.offsets.enabled`, then the
> > MirrorCheckpointTask will sync the offsets if all of the following is
> > true:
> > 1. The source group exists
> > 2. The source group name matches the configured group filter 
> > (group.filter.class, groups, groups.exclude) 3. The source group has an 
> > offset for a topic which matches the configured topic filter 
> > (topic.filter.class, topics, topics.exclude) 4. The target group does not 
> > exist, or has no active consumers 5. The target group has no offset for a 
> > specified partition, or the offset is earlier than the translated offset 6. 
> > MirrorCheckpointTask restarted after replication happened but before the 
> > offset could be translated.
> >
> > If one of these isn't true, you won't see translation happening.
> >
> > Are you having a problem with too many consumer groups being created?
> > You can restrict the group or topic filters, as they're very permissive by 
> > default.
> > Or is the problem that you expect the consumer groups to be created, but 
> > they aren't? One of the above conditions may not be satisfied, or your 
> > MirrorCheckpointConnector/Task may be malfunctioning and you'll need to 
> > inspect the logs to debug it.
> >
> > Best,
> > Greg Harris
> >
> > On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) 
> >  wrote:
> > >
> > > Hi
> > > Can someone advise please
> > > if sync.group.offsets.enabled : true to sync offset from source to
> > > target for particular consumer group That group must be created on 
> > > target, even if no actual consumption will be done ?
> > >
> > >
> > >
> > > This email and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs Email Terms of Service, which
> > > you may review at
> > > https://www.amdocs.com/about/email-terms-of-service
> > > <https://www.amdocs.com/about/email-terms-of-service>
> > This email and the information contained herein is proprietary and
> > confidential and subject to the Amdocs Email Terms of Service, which
> > you may review at https://www.amdocs.com/about/email-terms-of-service
> > <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and 
> confidential and subject to the Amdocs Email Terms of Service, which you may 
> review at https://www.amdocs.com/about/email-terms-of-service 
> <https://www.amdocs.com/about/email-terms-of-service>

RE: Mirror Maker 2 - offset sync from source to target

[Alexander Shapiro (ashapiro)]

Thanks, one clarification plz

In bullet for You mention "4. The target group does not exist, or has no active 
consumers"
If group on target does not exist, will it be created without active consumers ?

-Original Message-
From: Greg Harris  
Sent: Monday, October 23, 2023 8:56 PM
To: users@kafka.apache.org
Subject: Re: Mirror Maker 2 - offset sync from source to target

[You don't often get email from greg.har...@aiven.io.invalid. Learn why this is 
important at https://aka.ms/LearnAboutSenderIdentification ]

CAUTION: This email is from an external source. Please don’t open any unknown 
links or attachments.

Hi Alexander,

Sorry I noticed an inconsistency in my last email. For point 6:
If the MirrorCheckpointTask restarts after replication but before offset is 
translated, then it may not be able to perform a translation.
If the MirrorCheckpointTask does not restart, it should be able to perform 
translation.

So if your MirrorMaker2 is restarting frequently, that may prevent consistent 
translation.

Thanks,
Greg

On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro) 
 wrote:
>
> Hi Greg,
> Thank you very much,
> it is the most detailed answer I would expect.
>
> -Original Message-
> From: Greg Harris 
> Sent: Monday, October 23, 2023 8:42 PM
> To: users@kafka.apache.org
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> CAUTION: This email is from an external source. Please don’t open any unknown 
> links or attachments.
>
> Hi Alexander,
>
> Thanks for using MirrorMaker2!
>
> If you turn on `sync.group.offsets.enabled`, then the 
> MirrorCheckpointTask will sync the offsets if all of the following is
> true:
> 1. The source group exists
> 2. The source group name matches the configured group filter 
> (group.filter.class, groups, groups.exclude) 3. The source group has an 
> offset for a topic which matches the configured topic filter 
> (topic.filter.class, topics, topics.exclude) 4. The target group does not 
> exist, or has no active consumers 5. The target group has no offset for a 
> specified partition, or the offset is earlier than the translated offset 6. 
> MirrorCheckpointTask restarted after replication happened but before the 
> offset could be translated.
>
> If one of these isn't true, you won't see translation happening.
>
> Are you having a problem with too many consumer groups being created?
> You can restrict the group or topic filters, as they're very permissive by 
> default.
> Or is the problem that you expect the consumer groups to be created, but they 
> aren't? One of the above conditions may not be satisfied, or your 
> MirrorCheckpointConnector/Task may be malfunctioning and you'll need to 
> inspect the logs to debug it.
>
> Best,
> Greg Harris
>
> On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) 
>  wrote:
> >
> > Hi
> > Can someone advise please
> > if sync.group.offsets.enabled : true to sync offset from source to 
> > target for particular consumer group That group must be created on target, 
> > even if no actual consumption will be done ?
> >
> >
> >
> > This email and the information contained herein is proprietary and 
> > confidential and subject to the Amdocs Email Terms of Service, which 
> > you may review at 
> > https://www.amdocs.com/about/email-terms-of-service
> > <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and 
> confidential and subject to the Amdocs Email Terms of Service, which 
> you may review at https://www.amdocs.com/about/email-terms-of-service 
> <https://www.amdocs.com/about/email-terms-of-service>
This email and the information contained herein is proprietary and confidential 
and subject to the Amdocs Email Terms of Service, which you may review at 
https://www.amdocs.com/about/email-terms-of-service 
<https://www.amdocs.com/about/email-terms-of-service>

Re: Mirror Maker 2 - offset sync from source to target

[Greg Harris]

Hi Alexander,

Sorry I noticed an inconsistency in my last email. For point 6:
If the MirrorCheckpointTask restarts after replication but before
offset is translated, then it may not be able to perform a
translation.
If the MirrorCheckpointTask does not restart, it should be able to
perform translation.

So if your MirrorMaker2 is restarting frequently, that may prevent
consistent translation.

Thanks,
Greg

On Mon, Oct 23, 2023 at 10:46 AM Alexander Shapiro (ashapiro)
 wrote:
>
> Hi Greg,
> Thank you very much,
> it is the most detailed answer I would expect.
>
> -Original Message-
> From: Greg Harris 
> Sent: Monday, October 23, 2023 8:42 PM
> To: users@kafka.apache.org
> Subject: Re: Mirror Maker 2 - offset sync from source to target
>
> CAUTION: This email is from an external source. Please don’t open any unknown 
> links or attachments.
>
> Hi Alexander,
>
> Thanks for using MirrorMaker2!
>
> If you turn on `sync.group.offsets.enabled`, then the MirrorCheckpointTask 
> will sync the offsets if all of the following is
> true:
> 1. The source group exists
> 2. The source group name matches the configured group filter 
> (group.filter.class, groups, groups.exclude) 3. The source group has an 
> offset for a topic which matches the configured topic filter 
> (topic.filter.class, topics, topics.exclude) 4. The target group does not 
> exist, or has no active consumers 5. The target group has no offset for a 
> specified partition, or the offset is earlier than the translated offset 6. 
> MirrorCheckpointTask restarted after replication happened but before the 
> offset could be translated.
>
> If one of these isn't true, you won't see translation happening.
>
> Are you having a problem with too many consumer groups being created?
> You can restrict the group or topic filters, as they're very permissive by 
> default.
> Or is the problem that you expect the consumer groups to be created, but they 
> aren't? One of the above conditions may not be satisfied, or your 
> MirrorCheckpointConnector/Task may be malfunctioning and you'll need to 
> inspect the logs to debug it.
>
> Best,
> Greg Harris
>
> On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) 
>  wrote:
> >
> > Hi
> > Can someone advise please
> > if sync.group.offsets.enabled : true to sync offset from source to
> > target for particular consumer group That group must be created on target, 
> > even if no actual consumption will be done ?
> >
> >
> >
> > This email and the information contained herein is proprietary and
> > confidential and subject to the Amdocs Email Terms of Service, which
> > you may review at https://www.amdocs.com/about/email-terms-of-service
> > <https://www.amdocs.com/about/email-terms-of-service>
> This email and the information contained herein is proprietary and 
> confidential and subject to the Amdocs Email Terms of Service, which you may 
> review at https://www.amdocs.com/about/email-terms-of-service 
> <https://www.amdocs.com/about/email-terms-of-service>

RE: Mirror Maker 2 - offset sync from source to target

[Alexander Shapiro (ashapiro)]

Hi Greg,
Thank you very much, 
it is the most detailed answer I would expect.

-Original Message-
From: Greg Harris  
Sent: Monday, October 23, 2023 8:42 PM
To: users@kafka.apache.org
Subject: Re: Mirror Maker 2 - offset sync from source to target

CAUTION: This email is from an external source. Please don’t open any unknown 
links or attachments.

Hi Alexander,

Thanks for using MirrorMaker2!

If you turn on `sync.group.offsets.enabled`, then the MirrorCheckpointTask will 
sync the offsets if all of the following is
true:
1. The source group exists
2. The source group name matches the configured group filter 
(group.filter.class, groups, groups.exclude) 3. The source group has an offset 
for a topic which matches the configured topic filter (topic.filter.class, 
topics, topics.exclude) 4. The target group does not exist, or has no active 
consumers 5. The target group has no offset for a specified partition, or the 
offset is earlier than the translated offset 6. MirrorCheckpointTask restarted 
after replication happened but before the offset could be translated.

If one of these isn't true, you won't see translation happening.

Are you having a problem with too many consumer groups being created?
You can restrict the group or topic filters, as they're very permissive by 
default.
Or is the problem that you expect the consumer groups to be created, but they 
aren't? One of the above conditions may not be satisfied, or your 
MirrorCheckpointConnector/Task may be malfunctioning and you'll need to inspect 
the logs to debug it.

Best,
Greg Harris

On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro) 
 wrote:
>
> Hi
> Can someone advise please
> if sync.group.offsets.enabled : true to sync offset from source to 
> target for particular consumer group That group must be created on target, 
> even if no actual consumption will be done ?
>
>
>
> This email and the information contained herein is proprietary and 
> confidential and subject to the Amdocs Email Terms of Service, which 
> you may review at https://www.amdocs.com/about/email-terms-of-service 
> <https://www.amdocs.com/about/email-terms-of-service>
This email and the information contained herein is proprietary and confidential 
and subject to the Amdocs Email Terms of Service, which you may review at 
https://www.amdocs.com/about/email-terms-of-service 
<https://www.amdocs.com/about/email-terms-of-service>

Re: Mirror Maker 2 - offset sync from source to target

[Greg Harris]

Hi Alexander,

Thanks for using MirrorMaker2!

If you turn on `sync.group.offsets.enabled`, then the
MirrorCheckpointTask will sync the offsets if all of the following is
true:
1. The source group exists
2. The source group name matches the configured group filter
(group.filter.class, groups, groups.exclude)
3. The source group has an offset for a topic which matches the
configured topic filter (topic.filter.class, topics, topics.exclude)
4. The target group does not exist, or has no active consumers
5. The target group has no offset for a specified partition, or the
offset is earlier than the translated offset
6. MirrorCheckpointTask restarted after replication happened but
before the offset could be translated.

If one of these isn't true, you won't see translation happening.

Are you having a problem with too many consumer groups being created?
You can restrict the group or topic filters, as they're very
permissive by default.
Or is the problem that you expect the consumer groups to be created,
but they aren't? One of the above conditions may not be satisfied, or
your MirrorCheckpointConnector/Task may be malfunctioning and you'll
need to inspect the logs to debug it.

Best,
Greg Harris

On Sun, Oct 22, 2023 at 5:36 AM Alexander Shapiro (ashapiro)
 wrote:
>
> Hi
> Can someone advise please
> if sync.group.offsets.enabled : true to sync offset from source to target for 
> particular consumer group
> That group must be created on target, even if no actual consumption will be 
> done ?
>
>
>
> This email and the information contained herein is proprietary and 
> confidential and subject to the Amdocs Email Terms of Service, which you may 
> review at https://www.amdocs.com/about/email-terms-of-service 
>