Re: [DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-25 Thread Jorge Esteban Quilcate Otoya 
Hi Abhijeet,

Thanks for this KIP, I pretty much agree with the renaming and new names
look good to me.

Cheers,
Jorge.

On Tue, 25 Jul 2023 at 12:56, Satish Duggana 
wrote:

> Hi Abhijeet,
> Thanks for keeping this KIP only to renaming the existing metrics for
> better clarity. These new names look good to me.
>
> ~Satish.
>
> On Tue, 25 Jul 2023 at 13:12, Luke Chen  wrote:
> >
> > Hi Abhijeet,
> >
> > Thanks for the KIP!
> > I don't have much preference for the name changing.
> > But if it could confuse other people, it's good to make it clear.
> >
> > Thank you.
> > Luke
> >
> > On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> > wrote:
> >
> > > Hi Kamal,
> > >
> > > As we discussed offline, I will rename this KIP so that it only
> captures
> > > the aspect of renaming the previously added metrics to remove
> ambiguity.
> > > I will create another KIP for RemoteIndexCache metrics and other
> relevant
> > > tiered storage metrics.
> > >
> > > On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi Abhijeet,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > We are changing the metric names from what was proposed in the
> KIP-405
> > > and
> > > > adding new metrics for RemoteIndexCache.
> > > > In the KIP, it's not clear whether we are renaming the aggregate
> broker
> > > > level metrics for remote copy/fetch/failed-copy/failed-fetch.
> > > >
> > > > Are these metrics enough to monitor all the aspects of tiered
> storage?
> > > >
> > > > (eg)
> > > > 1. Metrics to see the Tier Lag Status by number of pending
> > > > segments/records.
> > > > 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> > > > expose local-log-start-offset and
> > > highest-offset-uploaded-to-remote-storage
> > > > as metric?
> > > >
> > > > Thanks,
> > > > Kamal
> > > >
> > > > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar <
> > > abhijeet.cse@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I created KIP-930 for adding RemoteIndexCache stats and also to
> rename
> > > > some
> > > > > tiered storage metrics added as part of KIP-405
> > > > > <
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > > > > >
> > > > > to remove ambiguity.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > > > >
> > > > > Feedback and suggestions are welcome.
> > > > >
> > > > > Regards,
> > > > > Abhijeet.
> > > > >
> > > >
> > >
> > >
> > > --
> > > Abhijeet.
> > >
>

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-25 Thread Satish Duggana 
Hi Abhijeet,
Thanks for keeping this KIP only to renaming the existing metrics for
better clarity. These new names look good to me.

~Satish.

On Tue, 25 Jul 2023 at 13:12, Luke Chen  wrote:
>
> Hi Abhijeet,
>
> Thanks for the KIP!
> I don't have much preference for the name changing.
> But if it could confuse other people, it's good to make it clear.
>
> Thank you.
> Luke
>
> On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar 
> wrote:
>
> > Hi Kamal,
> >
> > As we discussed offline, I will rename this KIP so that it only captures
> > the aspect of renaming the previously added metrics to remove ambiguity.
> > I will create another KIP for RemoteIndexCache metrics and other relevant
> > tiered storage metrics.
> >
> > On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi Abhijeet,
> > >
> > > Thanks for the KIP!
> > >
> > > We are changing the metric names from what was proposed in the KIP-405
> > and
> > > adding new metrics for RemoteIndexCache.
> > > In the KIP, it's not clear whether we are renaming the aggregate broker
> > > level metrics for remote copy/fetch/failed-copy/failed-fetch.
> > >
> > > Are these metrics enough to monitor all the aspects of tiered storage?
> > >
> > > (eg)
> > > 1. Metrics to see the Tier Lag Status by number of pending
> > > segments/records.
> > > 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> > > expose local-log-start-offset and
> > highest-offset-uploaded-to-remote-storage
> > > as metric?
> > >
> > > Thanks,
> > > Kamal
> > >
> > > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar <
> > abhijeet.cse@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I created KIP-930 for adding RemoteIndexCache stats and also to rename
> > > some
> > > > tiered storage metrics added as part of KIP-405
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > > > >
> > > > to remove ambiguity.
> > > >
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Regards,
> > > > Abhijeet.
> > > >
> > >
> >
> >
> > --
> > Abhijeet.
> >

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-25 Thread Luke Chen 
Hi Abhijeet,

Thanks for the KIP!
I don't have much preference for the name changing.
But if it could confuse other people, it's good to make it clear.

Thank you.
Luke

On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar 
wrote:

> Hi Kamal,
>
> As we discussed offline, I will rename this KIP so that it only captures
> the aspect of renaming the previously added metrics to remove ambiguity.
> I will create another KIP for RemoteIndexCache metrics and other relevant
> tiered storage metrics.
>
> On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Abhijeet,
> >
> > Thanks for the KIP!
> >
> > We are changing the metric names from what was proposed in the KIP-405
> and
> > adding new metrics for RemoteIndexCache.
> > In the KIP, it's not clear whether we are renaming the aggregate broker
> > level metrics for remote copy/fetch/failed-copy/failed-fetch.
> >
> > Are these metrics enough to monitor all the aspects of tiered storage?
> >
> > (eg)
> > 1. Metrics to see the Tier Lag Status by number of pending
> > segments/records.
> > 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> > expose local-log-start-offset and
> highest-offset-uploaded-to-remote-storage
> > as metric?
> >
> > Thanks,
> > Kamal
> >
> > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com
> > >
> > wrote:
> >
> > > Hi All,
> > >
> > > I created KIP-930 for adding RemoteIndexCache stats and also to rename
> > some
> > > tiered storage metrics added as part of KIP-405
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > > >
> > > to remove ambiguity.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Regards,
> > > Abhijeet.
> > >
> >
>
>
> --
> Abhijeet.
>

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-25 Thread Abhijeet Kumar 
Hi Kamal,

As we discussed offline, I will rename this KIP so that it only captures
the aspect of renaming the previously added metrics to remove ambiguity.
I will create another KIP for RemoteIndexCache metrics and other relevant
tiered storage metrics.

On Tue, Jul 25, 2023 at 12:03 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Abhijeet,
>
> Thanks for the KIP!
>
> We are changing the metric names from what was proposed in the KIP-405 and
> adding new metrics for RemoteIndexCache.
> In the KIP, it's not clear whether we are renaming the aggregate broker
> level metrics for remote copy/fetch/failed-copy/failed-fetch.
>
> Are these metrics enough to monitor all the aspects of tiered storage?
>
> (eg)
> 1. Metrics to see the Tier Lag Status by number of pending
> segments/records.
> 2. Similar to log-start-offset and log-end-offset metrics.  Should we
> expose local-log-start-offset and highest-offset-uploaded-to-remote-storage
> as metric?
>
> Thanks,
> Kamal
>
> On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar  >
> wrote:
>
> > Hi All,
> >
> > I created KIP-930 for adding RemoteIndexCache stats and also to rename
> some
> > tiered storage metrics added as part of KIP-405
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> > >
> > to remove ambiguity.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
> >
> > Feedback and suggestions are welcome.
> >
> > Regards,
> > Abhijeet.
> >
>


-- 
Abhijeet.

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-25 Thread Kamal Chandraprakash 
Hi Abhijeet,

Thanks for the KIP!

We are changing the metric names from what was proposed in the KIP-405 and
adding new metrics for RemoteIndexCache.
In the KIP, it's not clear whether we are renaming the aggregate broker
level metrics for remote copy/fetch/failed-copy/failed-fetch.

Are these metrics enough to monitor all the aspects of tiered storage?

(eg)
1. Metrics to see the Tier Lag Status by number of pending segments/records.
2. Similar to log-start-offset and log-end-offset metrics.  Should we
expose local-log-start-offset and highest-offset-uploaded-to-remote-storage
as metric?

Thanks,
Kamal

On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar 
wrote:

> Hi All,
>
> I created KIP-930 for adding RemoteIndexCache stats and also to rename some
> tiered storage metrics added as part of KIP-405
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
> >
> to remove ambiguity.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics
>
> Feedback and suggestions are welcome.
>
> Regards,
> Abhijeet.
>

[DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-24 Thread Abhijeet Kumar 
Hi All,

I created KIP-930 for adding RemoteIndexCache stats and also to rename some
tiered storage metrics added as part of KIP-405

to remove ambiguity.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics

Feedback and suggestions are welcome.

Regards,
Abhijeet.

[DISCUSS] KIP-930: Tiered Storage Metrics

2023-07-23 Thread Abhijeet Kumar 
Hi All,

I created KIP-930 for adding RemoteIndexCache stats and also to rename some
tiered storage metrics added as part of KIP-405

to
remove ambiguity.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics

Feedback and suggestions are welcome.

Regards,
Abhijeet.