[jira] [Commented] (KAFKA-3753) Metrics for StateStores
[ https://issues.apache.org/jira/browse/KAFKA-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322683#comment-15322683 ] Jeff Klukas commented on KAFKA-3753: Since there's refactoring going on right now with the metrics interface for streams (https://issues.apache.org/jira/browse/KAFKA-3715), I think we should delay actually adding size metrics to a different issue. The PR I attached here adds the size() method so that it can be used for a metric in the future. > Metrics for StateStores > --- > > Key: KAFKA-3753 > URL: https://issues.apache.org/jira/browse/KAFKA-3753 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Guozhang Wang >Priority: Minor > Labels: api > Fix For: 0.10.1.0 > > > As a developer building a Kafka Streams application, I'd like to have > visibility into what's happening with my state stores. How can I know if a > particular store is growing large? How can I know if a particular store is > frequently needing to hit disk? > I'm interested to know if there are existing mechanisms for extracting this > information or if other people have thoughts on how we might approach this. > I can't think of a way to provide metrics generically, so each state store > implementation would likely need to handle this separately. Given that the > default RocksDBStore will likely be the most-used, it would be a first target > for adding metrics. > I'd be interested in knowing the total number of entries in the store, the > total size on disk and in memory, rates of gets and puts, and hit/miss ratio > for the MemoryLRUCache. Some of these numbers are likely calculable through > the RocksDB API, others may simply not be accessible. > Would there be value to the wider community in having state stores register > metrics? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3753) Metrics for StateStores
[ https://issues.apache.org/jira/browse/KAFKA-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322678#comment-15322678 ] ASF GitHub Bot commented on KAFKA-3753: --- GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/1486 KAFKA-3753: Add size() method to KeyValueStore interface See https://issues.apache.org/jira/browse/KAFKA-3753 This contribution is my original work and I license the work to the project under the project's open source license. cc @guozhangwang @kichristensen @ijuma You can merge this pull request into a Git repository by running: $ git pull https://github.com/jklukas/kafka kvstore-size Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1486.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1486 commit c4ea7cd0ed4d822e65cee3fd283f81c29ce11d57 Author: Jeff Klukas Date: 2016-06-09T15:08:33Z Add size() method to KeyValueStore interface > Metrics for StateStores > --- > > Key: KAFKA-3753 > URL: https://issues.apache.org/jira/browse/KAFKA-3753 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Guozhang Wang >Priority: Minor > Labels: api > Fix For: 0.10.1.0 > > > As a developer building a Kafka Streams application, I'd like to have > visibility into what's happening with my state stores. How can I know if a > particular store is growing large? How can I know if a particular store is > frequently needing to hit disk? > I'm interested to know if there are existing mechanisms for extracting this > information or if other people have thoughts on how we might approach this. > I can't think of a way to provide metrics generically, so each state store > implementation would likely need to handle this separately. Given that the > default RocksDBStore will likely be the most-used, it would be a first target > for adding metrics. > I'd be interested in knowing the total number of entries in the store, the > total size on disk and in memory, rates of gets and puts, and hit/miss ratio > for the MemoryLRUCache. Some of these numbers are likely calculable through > the RocksDB API, others may simply not be accessible. > Would there be value to the wider community in having state stores register > metrics? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3753) Metrics for StateStores
[ https://issues.apache.org/jira/browse/KAFKA-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321511#comment-15321511 ] Guozhang Wang commented on KAFKA-3753: -- Yeah it's a good idea to add "size" to the KeyValueStore interface. > Metrics for StateStores > --- > > Key: KAFKA-3753 > URL: https://issues.apache.org/jira/browse/KAFKA-3753 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Guozhang Wang >Priority: Minor > Labels: api > Fix For: 0.10.1.0 > > > As a developer building a Kafka Streams application, I'd like to have > visibility into what's happening with my state stores. How can I know if a > particular store is growing large? How can I know if a particular store is > frequently needing to hit disk? > I'm interested to know if there are existing mechanisms for extracting this > information or if other people have thoughts on how we might approach this. > I can't think of a way to provide metrics generically, so each state store > implementation would likely need to handle this separately. Given that the > default RocksDBStore will likely be the most-used, it would be a first target > for adding metrics. > I'd be interested in knowing the total number of entries in the store, the > total size on disk and in memory, rates of gets and puts, and hit/miss ratio > for the MemoryLRUCache. Some of these numbers are likely calculable through > the RocksDB API, others may simply not be accessible. > Would there be value to the wider community in having state stores register > metrics? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3753) Metrics for StateStores
[ https://issues.apache.org/jira/browse/KAFKA-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318521#comment-15318521 ] Jeff Klukas commented on KAFKA-3753: It's good to know about MeteredKeyValueStore. And holding off on cache-related metrics for the redesign sounds reasonable. It looks like the KeyValueStore interface does not provide any method related to number of entries (besides calling `all()` and iterating over all values, which doesn't seem reasonable), so there's no way for MeteredKeyValueStore to access the number of entries in the wrapped store. Could we consider adding a `size` method to the KeyValueStore interface? > Metrics for StateStores > --- > > Key: KAFKA-3753 > URL: https://issues.apache.org/jira/browse/KAFKA-3753 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Jeff Klukas >Assignee: Guozhang Wang >Priority: Minor > Labels: api > Fix For: 0.10.1.0 > > > As a developer building a Kafka Streams application, I'd like to have > visibility into what's happening with my state stores. How can I know if a > particular store is growing large? How can I know if a particular store is > frequently needing to hit disk? > I'm interested to know if there are existing mechanisms for extracting this > information or if other people have thoughts on how we might approach this. > I can't think of a way to provide metrics generically, so each state store > implementation would likely need to handle this separately. Given that the > default RocksDBStore will likely be the most-used, it would be a first target > for adding metrics. > I'd be interested in knowing the total number of entries in the store, the > total size on disk and in memory, rates of gets and puts, and hit/miss ratio > for the MemoryLRUCache. Some of these numbers are likely calculable through > the RocksDB API, others may simply not be accessible. > Would there be value to the wider community in having state stores register > metrics? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3753) Metrics for StateStores
[
https://issues.apache.org/jira/browse/KAFKA-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305100#comment-15305100
]
Guozhang Wang commented on KAFKA-3753:
--
This is a good suggestion, and we can either implement it directly inside
{{MeteredKeyValueStore}} that wraps all key-value stores (it already have put /
get rates that you are interested), that can periodically query the wrapped
store for number of entries (for example see {{availableBytes}} in
{{RecordAccumulator}}).
As for cache related metrics, there is already a JIRA ticket:
https://issues.apache.org/jira/browse/KAFKA-3183, but note that we are
refactoring the caching layer soon as part of our trigger design. So we may
want to hold on that end.
> Metrics for StateStores
> ---
>
> Key: KAFKA-3753
> URL: https://issues.apache.org/jira/browse/KAFKA-3753
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Jeff Klukas
>Assignee: Guozhang Wang
>Priority: Minor
> Labels: api
> Fix For: 0.10.1.0
>
>
> As a developer building a Kafka Streams application, I'd like to have
> visibility into what's happening with my state stores. How can I know if a
> particular store is growing large? How can I know if a particular store is
> frequently needing to hit disk?
> I'm interested to know if there are existing mechanisms for extracting this
> information or if other people have thoughts on how we might approach this.
> I can't think of a way to provide metrics generically, so each state store
> implementation would likely need to handle this separately. Given that the
> default RocksDBStore will likely be the most-used, it would be a first target
> for adding metrics.
> I'd be interested in knowing the total number of entries in the store, the
> total size on disk and in memory, rates of gets and puts, and hit/miss ratio
> for the MemoryLRUCache. Some of these numbers are likely calculable through
> the RocksDB API, others may simply not be accessible.
> Would there be value to the wider community in having state stores register
> metrics?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
