[
https://issues.apache.org/jira/browse/KAFKA-17895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896326#comment-17896326
]
Clément MATHIEU commented on KAFKA-17895:
-----------------------------------------
> My guess is that this wasn't exposed as a metric because it's only an
> approximation and has no guarantee of being correct.
As [~mjsax] said, it’s already exposed for rocksdb. Indeed, it’s an
approximation.
Personally, I mostly use in-memory state stores. But anyone using rocksdb can
decide if the approximation is good enough for their use case.
> It's called `estimate-num-keys` – should be the same thing?
It is; but it’s rocksdb specific. It would be great to have it exposed for
in-memory state stores too. Ideally, metric name would be the same for both
types of state stores. If this is difficult, having a different name for the
two types is better than not having the information or having to add the metric
yourself.
To give you some context; I use Micrometer to collect Kafka Streams metrics,
export them to a Prometheus and then set a rule to fire an alert if the store
size exceeds a certain size. It works well for use cases where state stores are
used to persist pending work item (ex. retries of failures). In steady state,
store size is mostly linear to input load. A growing size signals there is a
problem.
> Expose KeyValueStore's approximateNumEntries as a metric
> --------------------------------------------------------
>
> Key: KAFKA-17895
> URL: https://issues.apache.org/jira/browse/KAFKA-17895
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 3.8.1
> Reporter: Clément MATHIEU
> Priority: Minor
> Labels: needs-kip
>
> Tracking the evolution of a state store's size is often useful.
>
> For example, we often use state store to persist pending work and set alert
> on maximum size because it means the process is falling behind.
> While KafkaStreams exposes many generic state store related or RocksDB
> specificif metrics; is does not expose KeyValueStore#approximateNumEntries
> which is a key information.
> Is it an oversight or was it a deliberate choice?
> I would be great if this metrics could be added. I assume that both in-memory
> & rocksdb implementation of approximateNumEntries are fast enought to be used
> in metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
