Ryan Stull created KAFKA-19419:
----------------------------------
Summary: MaxMessageTimestamp JMX metric per (topic, partition)
Key: KAFKA-19419
URL: https://issues.apache.org/jira/browse/KAFKA-19419
Project: Kafka
Issue Type: Improvement
Components: metrics
Reporter: Ryan Stull
I'm trying to setup some monitoring for our Kafka cluster via JMX, and
specifically I want to monitor how long it's been since each (topic,partition)
was written to.
Currently it seems that there's no JMX metric that corresponds exactly to what
I'm looking for, which would basically be the max(timestamp) of messages on a
per partition basis.
I'm currently using a hacky solution where I monitor the `LogEndOffset` and
note when it changes as a proxy for what would be `MaxMessageTimestamp`, but
that doesn't show me accurately if a topic's max timestamp is far in the past,
since it might have been written to a long time ago before my monitoring
service started collecting data.
Another more robust solution would be to actually create consumers to read each
topic and keep track of the max timestamp, but that seems like overkill since
we have nearly a hundred topics to monitor and all I need is the timestamp
field, and it seems there's no way to configure a consumer to just read the
metadata of a message, so I would be reading a huge amount of extraneous data,
putting unnecessary load on the cluster.
So basically, if a JMX metric like:
`kafka.log:name=MaxMessageTimestamp,partition=*,topic=*,type=Log`
Could be added, that would greatly help this use case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
