Thanks Igor for the suggestions. I updated the KIP with some of them. > s/recored's offset/recorded offsets I actually mean record’s offset and not recorded offset. MirrorSourceConnector store the offset of the committed record on the destination cluster in an internal topic.
> 12. The use of the word "customers" seems a bit odd to me in this context. > Did you perhaps mean one of "use-cases", "users" or "operators”? Changed this to use-cases > 13. "They still left with part#1 of this feature which add cost to > the progress of their replication." > I'm unsure what this means. Do you mean to say that > MirrorCheckpointConnector is disabled but MirrorSourceConnector is not? > Could you also clarify where the additional cost comes from? Yes currently MM2 can disable step 2 of this feature by not running MirrorCheckpointConnector or by disabling emit.checkpoints.enabled and sync.group.offsets.enabled however as long as MirrorSourceConnector is running this 1st step of the feature will never be disabled and this feature is costing 1. Create internal topic 2. Latency as we need to produce/queue translated offsets > 14. This is probably more ignorance of mine: it doesn't seem obvious in > the KIP how increasing offset.lag.max to INT_MAX helps reduce latency. Based on my knowledge and understanding, offset.lag.max controls the size of queued offsets to be synced which are stored in `pendingOffsetSyncs`. Then 1. MM2 will try to send these periodically in `commitRecord` using "MirrorSourceTask ::firePendingOffsetSyncs”. However any as `commitRecord` isn’t blocker then any failed offset sync will be skipped. 2. To fix this then in `commit` the connector which is a blocker method will retry publish any skipped offsets. Now If we sit offset.lag.max to high value then translated queued offset wouldn’t be considered stale and as a result wouldn’t be added to `pendingOffsetSyncs` which what `firePendingOffsetSyncs` will try to commit. Hope this clarify it. > On 20 Apr 2024, at 09:41, Igor Soarez <i...@soarez.me> wrote: > > Hi Omnia, > > Thanks for this KIP. > > 11. These seem to me to be small misspellings, please double-check: > s/MM2 main features/MM2's main features > s/syncing consumer group offset/syncing consumer group offsets > s/relays/relies > s/recored's offset/recorded offsets > s/clusters without need for/clusters without the need for > s/creating internal topic./creating an internal topic. > s/This KIP propose that/This KIP proposes that > > 12. The use of the word "customers" seems a bit odd to me in this context. > Did you perhaps mean one of "use-cases", "users" or "operators"? > > 13. "They still left with part#1 of this feature which add cost to > the progress of their replication." > I'm unsure what this means. Do you mean to say that > MirrorCheckpointConnector is disabled but MirrorSourceConnector is not? > Could you also clarify where the additional cost comes from? > > 14. This is probably more ignorance of mine: it doesn't seem obvious in > the KIP how increasing offset.lag.max to INT_MAX helps reduce latency. > I'm guessing it's related to KAFKA-14610 but after having a look I > still couldn't understand why. > > > -- > Igor > > On Wed, Apr 17, 2024, at 3:22 PM, Omnia Ibrahim wrote: >> Thanks Chris and Mickael for the votes. >> Can I please get one last +1 binding vote please? >> >> Thanks >> Omnia >> >>> On 12 Apr 2024, at 13:21, Chris Egerton <fearthecel...@gmail.com> wrote: >>> >>> +1 (binding), thanks Omnia! >>> >>> On Fri, Apr 12, 2024, 03:46 Mickael Maison <mickael.mai...@gmail.com> wrote: >>> >>>> Hi Omnia, >>>> >>>> +1 (binding), thanks for the KIP! >>>> >>>> Mickael >>>> >>>> On Fri, Apr 12, 2024 at 9:01 AM Omnia Ibrahim <o.g.h.ibra...@gmail.com> >>>> wrote: >>>>> >>>>> Hi everyone, I would like to start a voting thread for KIP-1031: Control >>>> offset translation in MirrorSourceConnector >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector >>>>> >>>>> For comments or feedback please check the discussion thread here >>>> https://lists.apache.org/thread/ym6zr0wrhglft5c000x9c8ych098s7h6 >>>>> >>>>> Thanks >>>>> Omnia >>>>> >>>> >> >>
