I have enabled yammer's ConsoleReporter and I am getting all the metrics
(including per-topic metrics).
Yammer's MetricName object implements equals/hashcode methods using
mBeanName . We are constructing a unique mBeanName for each metric, So we
are not missing/overwriting any metrics.
Current confusion is due to MetricName.name(). This will be same
(BytesInPerSec) for both broker level and topic level metrics. We need to
use MetricName.getMBeanName() to differentiate between broker level and
topic level metrics.
0.8.1 MBeanName:
"kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec"
"kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec"
0.8.2 MBeanName:
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC
ConsoleReporter's O/P:
BytesInPerSec: <- This is broker level
count = 1521
mean rate = 3.63 bytes/s
1-minute rate = 0.35 bytes/s
5-minute rate = 2.07 bytes/s
15-minute rate = 1.25 bytes/s
BytesInPerSec: <- This is for topic1
count = 626
mean rate = 1.89 bytes/s
1-minute rate = 0.42 bytes/s
5-minute rate = 31.53 bytes/s
15-minute rate = 64.66 bytes/s
BytesInPerSec: <- This is for topic2
count = 895
mean rate = 3.62 bytes/s
1-minute rate = 1.39 bytes/s
5-minute rate = 30.08 bytes/s
15-minute rate = 50.27 bytes/s
Manikumar
On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <j...@squareup.com> wrote:
> Ok,
>
> It looks like the yammer MetricName is not being created correctly for the
> sub metrics that include a topic. E.g. a metric with an mbeanName like:
>
> kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic
>
> appears to be malformed. A yammer MetricName has 4 fields that are used in
> creating a graphite metric, that are included in the constructor:
> group, type, name, scope.
>
> In this case, the metric with the above mbeanName has these fields set in
> the MetricName:
>
> group: "kafka.server"
> type: "BrokerTopicMetrics"
> name: "BytesInPerSec"
> scope: null
>
> Thus, the topic metrics all look the same, and get lumped into the
> top-level BrokerTopicMetrics (and thus that will now be double counted). It
> looks like the fix for kafka-1481 was where things got broken. It seems to
> have introduced ‘tags’ in the building of metric names, and then those tags
> only get applied to the mbeanName, but get excluded from the metric name:
>
> https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f
>
> This is a pretty severe issue, since the yammer metrics for these stats
> will be double counted in aggregate, and the per-topic stats will be
> removed.
>
> I should note too, in my previous email, I thought that only the per-topic
> BrokerTopicMetrics were missing, but also several other per-topic metrics
> are missing too, e.g. under kafka.log, etc.
>
> Jason
>
>
> On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <j...@squareup.com> wrote:
>
> > I can confirm that the per topic metrics are not coming through to the
> > yammer metrics registry. I do see them in jmx (via jconsole), but the
> > MetricsRegistry does not have them.
> > All the other metrics are coming through that appear in jmx.
> >
> > This is with single node instance running locally.
> >
> > Jason
> >
> >
> >
> > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy <ku...@nmsworks.co.in>
> > wrote:
> >
> >> If you are using multi-node cluster, then metrics may be reported from
> >> other servers.
> >> pl check all the servers in the cluster.
> >>
> >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <kyleban...@gmail.com>
> >> wrote:
> >>
> >> > I've been using a custom KafkaMetricsReporter to report Kafka broker
> >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and
> >> messages in
> >> > and out for all topics together and for each individual topic.
> >> >
> >> > After upgrading to v0.8.2.0, these metrics are no longer being
> reported.
> >> >
> >> > I'm only seeing the following:
> >> > BrokerTopicMetrics
> >> > - BytesInPerSec
> >> > - BytesOutPerSec
> >> > - BytesRejectedPerSec
> >> > - MessagesInPerSec
> >> >
> >> > What's more, despite lots of successful writes to the cluster, the
> >> values
> >> > for these remaining metrics are all zero.
> >> >
> >> > I saw that there was some refactoring of metric naming code. Was the
> >> > behavior supposed to have changed?
> >> >
> >> > Many thanks in advance.
> >> >
> >>
> >
> >
>
