Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hey Mickael, Just checking to see if you have any thoughts on this. thanks! On Thu, 11 Apr 2024 at 15:11, Elxan Eminov wrote: > Hi Mickael! > Any thoughts on this? > Thanks! > > On Wed, 3 Apr 2024 at 13:21, Elxan Eminov wrote: > >> Hi Mickael, >> Thanks for your response and apologies for a

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Mickael! Any thoughts on this? Thanks! On Wed, 3 Apr 2024 at 13:21, Elxan Eminov wrote: > Hi Mickael, > Thanks for your response and apologies for a huge delay in mine. > > My thinking is that any partition could go stale if there are no records > being produced into it. If enough of such

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Mickael, Thanks for your response and apologies for a huge delay in mine. My thinking is that any partition could go stale if there are no records being produced into it. If enough of such partitions are present and are owned by a single MM task, an OOM could happen. Regarding the scenario

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi, Thanks for the updates. I'm wondering whether we really need the ttl eviction mechanism. The motivation is to "avoid storing stale LRO entries which can cause an eventual OOM error". How could it contain stake entries? I would expect its cache to only contain entries for partitions assigned

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Mickael! Any further thoughts on this? Thanks, Elkhan On Thu, 18 Jan 2024 at 11:53, Mickael Maison wrote: > Hi Elxan, > > Thanks for the updates. > > We used dots to separate words in configuration names, so I think > replication.offset.lag.metric.last-replicated-offset.ttl should be >

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi All, Since I changed the name of the KIP, the prev. long link doesn't work anymore. Here is the shortlink: https://cwiki.apache.org/confluence/x/95OzDw Thanks On Fri, 19 Jan 2024 at 18:08, Elxan Eminov wrote: > Hi Mickael, > Thanks for your feedback. > > I've updated the metric name to

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Mickael, Thanks for your feedback. I've updated the metric name to "replication-record-lage" and I've modified the property names to use dots instead of hyphes. I'm curious to know what you think of the proposed default values, let me know! Thanks, Elkhan On Thu, 18 Jan 2024 at 11:53,

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Elxan, Thanks for the updates. We used dots to separate words in configuration names, so I think replication.offset.lag.metric.last-replicated-offset.ttl should be named replication.offset.lag.metric.last.replicated.offset.ttl instead. About the names of the metrics, fair enough if you

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Apologies, forgot to reply on your last comment about the metric name. I believe both replication-lag and record-lag are a little too abstract - what do you think about either leaving it as replication-offset-lag or renaming to replication-record-lag? Thanks On Wed, 10 Jan 2024 at 15:31, Mickael

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Mickael, Thanks for the response. I've added the proposed default value for the refresh interval config. I've also added a new config to the KIP that will enable a TTL-based eviction policy for the in-memory LRO cache, as @huqedi suggested - also added a proposed default for this configuration.

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Elxan, Thanks for the KIP, it looks like a useful addition. Can you add to the KIP the default value you propose for replication.lag.metric.refresh.interval? In MirrorMaker most interval configs can be set to -1 to disable them, will it be the case for this new feature or will this setting

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi all, Here is the vote thread: https://lists.apache.org/thread/ftlnolcrh858dry89sjg06mdcdj9mrqv Cheers! On Wed, 27 Dec 2023 at 11:23, Elxan Eminov wrote: > Hi all, > I've updated the KIP with the details we discussed in this thread. > I'll call in a vote after the holidays if everything

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi all, I've updated the KIP with the details we discussed in this thread. I'll call in a vote after the holidays if everything looks good. Thanks! On Sat, 26 Aug 2023 at 15:49, Elxan Eminov wrote: > Relatively minor change with a new metric for MM2 > >

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Viktor, Apologies, didn't get time to spare for this one. I will add the proposed config name to the KIP and propose a vote. Thanks! On Mon, 4 Dec 2023 at 16:14, Viktor Somogyi-Vass < viktor.somo...@cloudera.com> wrote: > Elkhan, do you want to propose a vote for this KIP or do you have any

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Elkhan, do you want to propose a vote for this KIP or do you have any other ideas to include? On Tue, Oct 17, 2023 at 2:47 PM Viktor Somogyi-Vass < viktor.somo...@cloudera.com> wrote: > Hi hudeqi, > > Good thinking about the OOM and resource leaks. > The "update.replication.lag.interval.time" I

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi hudeqi, Good thinking about the OOM and resource leaks. The "update.replication.lag.interval.time" I think is almost good but we should include that it is about a metric (like "replication.lag.interval.metric.update.time") so it's obvious without the docs too. Thanks, Viktor On Sat, Oct 7,

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi, Elkhan, Viktor. I took a look at the updated KIP. I think Viktor mentioned that he did not see the relevant configuration, which refers to "(Optional) - MirrorConnectorConfig - a configuration to control the poll interval for the Consumer.endOffsets() call at LEO acquisition mentioned

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Viktor, When you say "I don't see the config you introduced on the KIP" - do you mean that the potential config key name is not specified? Or are there any other specific details I need to add? If so, could you please link an example KIP that I can reference? Thanks alot! On Wed, 27 Sept 2023

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Elkhan, I don't see the config you introduced on the KIP. Otherwise it looks good to me. If no one has objections or anything to add then I think you can start the vote after updating the KIP. >I added the configurable poll interval as an optional - I believe it is useful to let users to

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Team, Just pinging this Thanks, Elkhan On Thu, 21 Sep 2023 at 23:09, Elxan Eminov wrote: > Hi Viktor and huqedi, > I've made the changes to the KIP. > I added the configurable poll interval as an optional - I believe it is > useful to let users to configure it but I doubt whether it would

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Viktor and huqedi, I've made the changes to the KIP. I added the configurable poll interval as an optional - I believe it is useful to let users to configure it but I doubt whether it would actually be used. Please let me know your thoughts on this, Thanks! On Sat, 26 Aug 2023 at 15:49, Elxan

Re: Re: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

; > > > > > > On Mon, 4 Sept 2023 at 09:36, Elxan Eminov < > elxanemino...@gmail.com> > > > > wrote: > > > > > > > > > > > I already have the PR for this so if it will make it easier to > > discuss, > > > > > > feel free to take a look: > > https://github.com/apache/kafka/pull/14077 > > > > > > > > > > > > On Mon, 4 Sept 2023 at 09:17, hudeqi <16120...@bjtu.edu.cn> > wrote: > > > > > > > > > > > >> But does the offset of the last `ConsumerRecord` obtained in > poll > > not > > > > > >> only represent the offset of this record in the source cluster? > It > > > > seems > > > > > >> that it cannot represent the LEO of the source cluster for this > > > > partition. > > > > > >> I understand that the offset lag introduced here should be the > > LEO of > > > > the > > > > > >> source cluster minus the offset of the last record to be polled? > > > > > >> > > > > > >> best, > > > > > >> hudeqi > > > > > >> > > > > > >> > > > > > >> -原始邮件- > > > > > >> 发件人: "Elxan Eminov" > > > > > >> 发送时间: 2023-09-04 14:52:08 (星期一) > > > > > >> 收件人: dev@kafka.apache.org > > > > > >> 抄送: > > > > > >> 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag > > > > MirrorMaker2 > > > > > >> metric > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > >

Re: Re: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

o > discuss, > > > > > feel free to take a look: > https://github.com/apache/kafka/pull/14077 > > > > > > > > > > On Mon, 4 Sept 2023 at 09:17, hudeqi <16120...@bjtu.edu.cn> wrote: > > > > > > > > > >> But does the offset of the last `ConsumerRecord` obtained in poll > not > > > > >> only represent the offset of this record in the source cluster? It > > > seems > > > > >> that it cannot represent the LEO of the source cluster for this > > > partition. > > > > >> I understand that the offset lag introduced here should be the > LEO of > > > the > > > > >> source cluster minus the offset of the last record to be polled? > > > > >> > > > > >> best, > > > > >> hudeqi > > > > >> > > > > >> > > > > >> -原始邮件- > > > > >> 发件人: "Elxan Eminov" > > > > >> 发送时间: 2023-09-04 14:52:08 (星期一) > > > > >> 收件人: dev@kafka.apache.org > > > > >> 抄送: > > > > >> 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag > > > MirrorMaker2 > > > > >> metric > > > > >> > > > > >> > > > > > > > > > > > > > >

Re: Re: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

the last `ConsumerRecord` obtained in poll not > > > >> only represent the offset of this record in the source cluster? It > > seems > > > >> that it cannot represent the LEO of the source cluster for this > > partition. > > > >> I understand that the offset lag introduced here should be the LEO of > > the > > > >> source cluster minus the offset of the last record to be polled? > > > >> > > > >> best, > > > >> hudeqi > > > >> > > > >> > > > >> -原始邮件- > > > >> 发件人: "Elxan Eminov" > > > >> 发送时间: 2023-09-04 14:52:08 (星期一) > > > >> 收件人: dev@kafka.apache.org > > > >> 抄送: > > > >> 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag > > MirrorMaker2 > > > >> metric > > > >> > > > >> > > > > > > > > > >

Re: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

> > >> only represent the offset of this record in the source cluster? It > seems > > >> that it cannot represent the LEO of the source cluster for this > partition. > > >> I understand that the offset lag introduced here should be the LEO of > the > > >> source cluster minus the offset of the last record to be polled? > > >> > > >> best, > > >> hudeqi > > >> > > >> > > >> -原始邮件- > > >> 发件人: "Elxan Eminov" > > >> 发送时间: 2023-09-04 14:52:08 (星期一) > > >> 收件人: dev@kafka.apache.org > > >> 抄送: > > >> 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag > MirrorMaker2 > > >> metric > > >> > > >> > > > > > > >

Re: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

t lag introduced here should be the LEO of the > >> source cluster minus the offset of the last record to be polled? > >> > >> best, > >> hudeqi > >> > >> > >> -原始邮件- > >> 发件人: "Elxan Eminov" > >> 发送时间: 2023-09-04 14:52:08 (星期一) > >> 收件人: dev@kafka.apache.org > >> 抄送: > >> 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 > >> metric > >> > >> > > > >

Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

件- >> 发件人: "Elxan Eminov" >> 发送时间: 2023-09-04 14:52:08 (星期一) >> 收件人: dev@kafka.apache.org >> 抄送: >> 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 >> metric >> >> > >

Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

d? > > best, > hudeqi > > > -原始邮件- > 发件人: "Elxan Eminov" > 发送时间: 2023-09-04 14:52:08 (星期一) > 收件人: dev@kafka.apache.org > 抄送: > 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 > metric > >

Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

收件人: dev@kafka.apache.org > 抄送: > 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 > metric > >

Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

of the source cluster minus the offset of the last record to be polled? best, hudeqi -原始邮件- 发件人: "Elxan Eminov" 发送时间: 2023-09-04 14:52:08 (星期一) 收件人: dev@kafka.apache.org 抄送: 主题: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

poll`? > > best, > hudeqi > > > -原始邮件- > 发件人: "Elxan Eminov" > 发送时间: 2023-09-02 19:18:23 (星期六) > 收件人: dev@kafka.apache.org > 抄送: > 主题: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag > MirrorMaker2 metric > >

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi, Eminov. My doubt is: how do you get the LEO of the source cluster for the partition in `poll`? best, hudeqi -原始邮件- 发件人: "Elxan Eminov" 发送时间: 2023-09-02 19:18:23 (星期六) 收件人: dev@kafka.apache.org 抄送: 主题: Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag Mi

Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi huqedi, Thanks for taking a look at this, The *poll* is executed by the task anyway (it's an essential part of the logic) and this KIP only aims to piggyback that request - it will not make any kind of additional requests, thus not adding any performance overhead. Thanks for mentioning the

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Viktor, Thanks for the feedback, 1. The min, max and avg variants are calculated automatically via the SampledStat abstraction -

Re: Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Elkhan, I have also done work similar to the partition replication lag mentioned in this kip. After going online, I discovered a problem: when `MirrorSourceTask` executes `poll`, it obtains the LEO of the source topic through the consumer. This logic may take seconds to jitter. , I think

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Hi Elkhan, I think this is quite a useful improvement. A few questions, suggestions: 1. How do you calculate the min, max and avg variants? If I understand correctly then the metric itself is partition based (where replication-offset-lag is the lag of the replica that is being consumed) and these

[DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

Relatively minor change with a new metric for MM2 https://cwiki.apache.org/confluence/display/KAFKA/KIP-971%3A+Expose+replication-offset-lag+MirrorMaker2+metric