Thanks for the KIP, Randall.
The KIP has one MBean per metric name. Can I suggest an alternate grouping?
kafka.connect:type=connector-metrics,connector=([-.\w]+)
connector-type
connector-class
connector-version
status
kafka.connect:type=task-metrics,connector=([-.\w]+),task=([\d]+)
status
pause-ratio
offset-commit-success-percentage
offset-commit-failure-percentage
offset-commit-max-time
offset-commit-99p-time
offset-commit-95p-time
offset-commit-90p-time
offset-commit-75p-time
offset-commit-50p-time
batch-size-max
batch-size-avg
kafka.connect:type=source-task-metrics,connector=([-.\w]+),task=([\d]+)
source-record-poll-rate
source-record-write-rate
kafka.connect:type=sink-task-metrics,connector=([-.\w]+),task=([\d]+)
sink-record-read-rate
sink-record-send-rate
sink-record-lag-max
partition-count
offset-commit-95p-time
offset-commit-90p-time
offset-commit-75p-time
offset-commit-50p-time
batch-size-max
batch-size-avg
kafka.connect:type=sink-task-metrics,connector=([-.\w]+),task=([\d]+),topic=([-.\w]+),partition=([\d]+)
sink-record-lag
sink-record-lag-avg
sink-record-lag-max
kafka.connect:type=connect-coordinator-metrics
task-count
connector-count
leader-name
state
rest-request-rate
kafka.connect:type=connect-coordinator-metrics,name=assigned-tasks
assigned-tasks (existing metric, so can't merge in above without
breaking compatibility)
kafka.connect:type=connect-coordinator-metrics,name=assigned-connectors
(existing metric, so can't merge in above without breaking compatibility)
assigned-connectors (existing metric, so can't merge in above without
breaking compatibility)
kafka.connect:type=connect-worker-rebalance-metrics
rebalance-success-total
rebalance-success-percentage
rebalance-failure-total
rebalance-failure-percentage
rebalance-max-time
rebalance-99p-time
rebalance-95p-time
rebalance-90p-time
rebalance-75p-time
rebalance-50p-time
time-since-last-rebalance
task-failure-rate
This lets you use a single MBean selector to select multiple related attributes
all at once. You can use JMX's wildcards to target which connectors or tasks or
topics or partitions you care about.
Also notice that for the topic and partition level metrics, the attributes are
named identically ("sink-record-lag-avg" instead of
"sink-record-{topic}-{partition}.records-lag-avg"), so monitoring systems have
a consistent string they can use, instead of needing to prefix-and-suffix
matching against the attribute name. And TBH, it integrates better with the
work I'm doing in https://issues.apache.org/jira/browse/KAFKA-3480
-James
> On Sep 7, 2017, at 4:50 PM, Randall Hauch <[email protected]> wrote:
>
> Hi everyone.
>
> I've created a new KIP to add metrics to the Kafka Connect framework:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
>
> The KIP approval deadline is looming, so if you're interested in Kafka
> Connect metrics please review and provide feedback as soon as possible. I'm
> interested not only in whether the metrics are sufficient and appropriate,
> but also in whether the MBean naming conventions are okay.
>
> Best regards,
>
> Randall
