Andrew, Yes, in 0.8.2.x, we cleaned up the metric name to put meaning attributes in tags, instead of in the name. This makes it easier for monitoring applications to parse those metric names. It seems that we need a CSVMetricsReporter that can deal with tagged names better. Many people also use jmxtrans to export the Kafka metrics.
Thanks, Jun On Mon, Dec 14, 2015 at 10:32 AM, Andrew Grasso <[email protected]> wrote: > Hello, > > I've been trying to monitor a Kafka broker using the CSVMetricsReporter. > However, when I start the broker, only a few csv files are created in the > directory, and then there are repeated IOExceptions in the kafka.out log > stating that CsvReporter.createStreamForMetric cannot create the file > {path_to_metrics_dir}/LocalTimeMs.csv > > I can recreate this problem in 0.8.2.1, 0.8.2.2, and 0.9.0.0. CSV reporting > works as expected in 0.8.1.1. > > I believe the problem is that in the later versions CSV files are being > named from only the metric name, and not the tags. In 0.8.1.1, the > LocalTimeMs metrics are given a name prefixed with a distinguishing source, > while in later versions this information is passed as a tag. These tags are > passed into the MetricName constructor in > KafkaMetricsGroup.explicitMetricName, but as the mBeanName and not as the > name. > > This is causing a name conflict, since CSV files are named off of the name > field, not the mBeanName. > > For a similar reason, I cannot get metrics on partition specific data. The > CSV reporter seems to not try any other metrics after raising an exception > on LocalTimeMs. All metrics are published correctly to JMX. > > Has anyone encountered this problem? Is there a way to make CSV metrics > work in the stable release (0.8.2.2)? > > It is possible I have missed something in following the code, as I am not > very literate in Scala, but I am reasonably confident that my understanding > of the code lines up with the behavior I've seen. > > I believe this behavior is related to an old issue, KAFKA-542 > <https://issues.apache.org/jira/browse/KAFKA-542> > > Best, > Andrew Grasso >
