[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742406#comment-17742406 ] Divij Vaidya commented on KAFKA-14953: -- Hey [~abhijeetkumar] [~jeqo] We should not change the public facing interfaces defined in the KIP after it has been voted and accepted. I observed that we are changing a few metric name such as [https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=97554472=364=363] but that would break the contract defined by the KIP. Let's create new KIPs for changes in the metrics where we can deprecate old ones and create new ones. > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741313#comment-17741313 ] Abhijeet Kumar commented on KAFKA-14953: [~jeqo] Adding the additional metric for `RemoteWriteRequestsPerSec` Also renaming the confusing metrics to "RemoteCopyBytes" and "RemoteFetchBytes" We can add the retention-related metrics in a follow-up diff. > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739661#comment-17739661 ] Jorge Esteban Quilcate Otoya commented on KAFKA-14953: -- Finally, it may be worth adding metrics for the retention related actions as well: `RemoteDeleteRequestsPerSec`, `RemoteDeleteErrorPerSec`, though we may add this within or after https://issues.apache.org/jira/browse/KAFKA-14888. > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739655#comment-17739655 ] Jorge Esteban Quilcate Otoya commented on KAFKA-14953: -- I also find confusing the semantics of `RemoteBytesIn|Out`: In regular Kafka, BytesIn means bytes written to the log, and BytesOut are bytes read from the log; but with TS: * RemoteBytesIn means "Number of bytes read from remote storage per second." and * RemoteBytesOut means "Number of bytes copied to remote storage per second." which is the opposite. Would it make sense to align these metric names? Another option may be to not trying to reuse the names, and use more specific metric names: "RemoteCopyBytes", "RemoteFetchBytes" instead? > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[
https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739647#comment-17739647
]
Jorge Esteban Quilcate Otoya commented on KAFKA-14953:
--
[~abhijeetkumar] thanks for working on this PR!
Looking into the metrics available, seems that we are missing
`{color:#00}RemoteWriteRequestsPerSec{color}` (there is
`{color:#00}RemoteWriteErrorPerSec{color}`and
`{color:#00}RemoteReadRequestsPerSec{color}`, I don't see why not adding
it). I think we can make a small amend to the KIP, and ship ti on the same PR,
wdyt? cc [~showuon]
> Add metrics for tiered storage
> --
>
> Key: KAFKA-14953
> URL: https://issues.apache.org/jira/browse/KAFKA-14953
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Divij Vaidya
>Assignee: Abhijeet Kumar
>Priority: Minor
>
> Not just for expired fetch. We also need to add all the metrics described in
> KIP-405
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics
>
> ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735298#comment-17735298 ] Jorge Esteban Quilcate Otoya commented on KAFKA-14953: -- [~showuon] I pressed add to soon, I got confused with the PR referred on the ticket description that is merged already, but it's only a reference. Anyway, thanks [~abhijeetkumar] for the update! > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735279#comment-17735279 ] Abhijeet Kumar commented on KAFKA-14953: I am working on the unit tests and the PR should be available soon. > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735219#comment-17735219 ] Luke Chen commented on KAFKA-14953: --- [~jeqo] , which PR are you referring to? We haven't got any metrics for tiered storage, yet. [~abhijeetkumar] is working on it. Do you need any help on it [~abhijeetkumar] ? > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735200#comment-17735200 ] Jorge Esteban Quilcate Otoya commented on KAFKA-14953: -- I think this one is merged, should we close this ticket? > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726550#comment-17726550 ] Luke Chen commented on KAFKA-14953: --- Sounds good! Thank you! > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726301#comment-17726301 ] Abhijeet Kumar commented on KAFKA-14953: Yes, that is in the plan as mentioned by Satish in the comment: [https://github.com/apache/kafka/pull/13535#discussion_r1181518655] I plan to add all the metrics in scope. > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14953) Add metrics for tiered storage
[ https://issues.apache.org/jira/browse/KAFKA-14953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726121#comment-17726121 ] Luke Chen commented on KAFKA-14953: --- [~abhijeetkumar] , I've expanded the scope of this ticket. Are you still interested in taking this ? > Add metrics for tiered storage > -- > > Key: KAFKA-14953 > URL: https://issues.apache.org/jira/browse/KAFKA-14953 > Project: Kafka > Issue Type: Sub-task >Reporter: Divij Vaidya >Assignee: Abhijeet Kumar >Priority: Minor > > Not just for expired fetch. We also need to add all the metrics described in > KIP-405 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-NewMetrics > > ref: [https://github.com/apache/kafka/pull/13535#discussion_r1180286031] -- This message was sent by Atlassian Jira (v8.20.10#820010)
