Thanks for the quick response, Jun (and many thanks to Jason for confirming and further investigating the issue). I've tested the patch, and it does fix the fundamental issue, but I do have a few comments that I'll leave on the ticket.
On Tue, Jan 27, 2015 at 9:19 AM, Jun Rao <j...@confluent.io> wrote: > Jason, Kyle, > > I created an 0.8.2 blocker > https://issues.apache.org/jira/browse/KAFKA-1902 > and attached a patch there. Could you test it out and see if it fixes the > issue with the reporter? The patch adds tags as scope in MetricName. > > Thanks, > > Jun > > On Tue, Jan 27, 2015 at 7:39 AM, Jun Rao <j...@confluent.io> wrote: > > > Jason, > > > > So, this sounds like a real issue. Perhaps we can fix it just by setting > > the tag name as the scope. For example, for mbean kafka.server:type= > > BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have > > > > group: "kafka.server" > > type: "BrokerTopicMetrics" > > name: "BytesInPerSec" > > scope: "topic=test" > > > > Do you know if scope can have characters like "=" and "," (e.g., for > scope > > like "topic=test,partition=1")? > > > > The issue with using mytopic-BytesInPerSec as the name is what we are > > trying to fix in kafka-1481. Topic name (and clientId, etc) can have dash > > in it and it's hard to parse. > > > > Thanks, > > > > Jun > > > > > > > > On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <j...@squareup.com> > wrote: > > > >> Remember multiple people have reported this issue. Per topic metrics no > >> longer appear in graphite (or in any system modeled after the yammer > >> GraphiteReporter). They are not being seen as unique. > >> > >> While these metrics are registered in the registry as separate > >> ‘MetricName’ > >> instances (varying only by mbeanName), the GraphiteReporter sends the > >> metrics to graphite using only the 4 fields I describe above. > >> Consequently, > >> multiple metrics in the registry get sent to graphite under the same > name. > >> Thus these metrics all end up in the same bucket in graphite, trampling > >> over each other making them useless. They aren’t ‘double counted’ so > much > >> as flapping between multiple independent values. > >> > >> We actually have our own Reporter class (based off the yammer > >> GraphiteReporter). Our version sends metrics through kafka which is then > >> consumed downstream by multiple metric consumers. > >> > >> The ConsoleReporter isn’t useful for actually persisting metrics > anywhere. > >> It’s just iterating through all the (identically named metrics in the > >> registry (save for the different mbeanNames))…. > >> > >> The mbeanName, as constructed, is not useful as a human readable metric > >> name, to be presented in a browsable tree of metrics, etc. The > >> ‘group’:’type’:’name’:’scope’ are the pieces that matter. > >> > >> The fix here is to produce MetricName instances similar to 0.8.1.1, etc. > >> In > >> this case, it should probably be something like: > >> > >> group: kafka.server > >> type: BrokerTopicMetrics > >> name: mytopic-BytesInPerSec > >> group: <unused> > >> > >> Jason > >> > >> > >> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in> > >> wrote: > >> > >> > I have enabled yammer's ConsoleReporter and I am getting all the > metrics > >> > (including per-topic metrics). > >> > > >> > Yammer's MetricName object implements equals/hashcode methods using > >> > mBeanName . We are constructing a unique mBeanName for each metric, So > >> we > >> > are not missing/overwriting any metrics. > >> > > >> > Current confusion is due to MetricName.name(). This will be same > >> > (BytesInPerSec) for both broker level and topic level metrics. We need > >> to > >> > use MetricName.getMBeanName() to differentiate between broker level > and > >> > topic level metrics. > >> > > >> > 0.8.1 MBeanName: > >> > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" > >> > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec" > >> > > >> > 0.8.2 MBeanName: > >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec > >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC > >> > > >> > > >> > ConsoleReporter's O/P: > >> > > >> > BytesInPerSec: <- This is broker level > >> > count = 1521 > >> > mean rate = 3.63 bytes/s > >> > 1-minute rate = 0.35 bytes/s > >> > 5-minute rate = 2.07 bytes/s > >> > 15-minute rate = 1.25 bytes/s > >> > > >> > BytesInPerSec: <- This is for topic1 > >> > count = 626 > >> > mean rate = 1.89 bytes/s > >> > 1-minute rate = 0.42 bytes/s > >> > 5-minute rate = 31.53 bytes/s > >> > 15-minute rate = 64.66 bytes/s > >> > > >> > BytesInPerSec: <- This is for topic2 > >> > count = 895 > >> > mean rate = 3.62 bytes/s > >> > 1-minute rate = 1.39 bytes/s > >> > 5-minute rate = 30.08 bytes/s > >> > 15-minute rate = 50.27 bytes/s > >> > > >> > Manikumar > >> > > >> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <j...@squareup.com> > >> wrote: > >> > > >> > > Ok, > >> > > > >> > > It looks like the yammer MetricName is not being created correctly > for > >> > the > >> > > sub metrics that include a topic. E.g. a metric with an mbeanName > >> like: > >> > > > >> > > > >> > > >> > kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic > >> > > > >> > > appears to be malformed. A yammer MetricName has 4 fields that are > >> used > >> > in > >> > > creating a graphite metric, that are included in the constructor: > >> > > group, type, name, scope. > >> > > > >> > > In this case, the metric with the above mbeanName has these fields > >> set in > >> > > the MetricName: > >> > > > >> > > group: "kafka.server" > >> > > type: "BrokerTopicMetrics" > >> > > name: "BytesInPerSec" > >> > > scope: null > >> > > > >> > > Thus, the topic metrics all look the same, and get lumped into the > >> > > top-level BrokerTopicMetrics (and thus that will now be double > >> counted). > >> > It > >> > > looks like the fix for kafka-1481 was where things got broken. It > >> seems > >> > to > >> > > have introduced ‘tags’ in the building of metric names, and then > those > >> > tags > >> > > only get applied to the mbeanName, but get excluded from the metric > >> name: > >> > > > >> > > > >> > > >> > https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f > >> > > > >> > > This is a pretty severe issue, since the yammer metrics for these > >> stats > >> > > will be double counted in aggregate, and the per-topic stats will be > >> > > removed. > >> > > > >> > > I should note too, in my previous email, I thought that only the > >> > per-topic > >> > > BrokerTopicMetrics were missing, but also several other per-topic > >> metrics > >> > > are missing too, e.g. under kafka.log, etc. > >> > > > >> > > Jason > >> > > > >> > > > >> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <j...@squareup.com> > >> > wrote: > >> > > > >> > > > I can confirm that the per topic metrics are not coming through to > >> the > >> > > > yammer metrics registry. I do see them in jmx (via jconsole), but > >> the > >> > > > MetricsRegistry does not have them. > >> > > > All the other metrics are coming through that appear in jmx. > >> > > > > >> > > > This is with single node instance running locally. > >> > > > > >> > > > Jason > >> > > > > >> > > > > >> > > > > >> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy < > >> ku...@nmsworks.co.in > >> > > > >> > > > wrote: > >> > > > > >> > > >> If you are using multi-node cluster, then metrics may be reported > >> from > >> > > >> other servers. > >> > > >> pl check all the servers in the cluster. > >> > > >> > >> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker < > kyleban...@gmail.com > >> > > >> > > >> wrote: > >> > > >> > >> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka > >> broker > >> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and > >> > > >> messages in > >> > > >> > and out for all topics together and for each individual topic. > >> > > >> > > >> > > >> > After upgrading to v0.8.2.0, these metrics are no longer being > >> > > reported. > >> > > >> > > >> > > >> > I'm only seeing the following: > >> > > >> > BrokerTopicMetrics > >> > > >> > - BytesInPerSec > >> > > >> > - BytesOutPerSec > >> > > >> > - BytesRejectedPerSec > >> > > >> > - MessagesInPerSec > >> > > >> > > >> > > >> > What's more, despite lots of successful writes to the cluster, > >> the > >> > > >> values > >> > > >> > for these remaining metrics are all zero. > >> > > >> > > >> > > >> > I saw that there was some refactoring of metric naming code. > Was > >> the > >> > > >> > behavior supposed to have changed? > >> > > >> > > >> > > >> > Many thanks in advance. > >> > > >> > > >> > > >> > >> > > > > >> > > > > >> > > > >> > > >> > > > > >
