Jason, Kyle, Added comments to the jira. Let me know if they make sense. The dot convention is a bit tricky to follow since we allow dots in topic and clientId, etc. Also, we probably don't want to use a convention too specific for Graphite since other systems may have other conventions.
Thanks, Jun On Tue, Jan 27, 2015 at 9:32 AM, Jason Rosenberg <j...@squareup.com> wrote: > I added a comment to the ticket. I think it will work getting data > disambiguated (didn't actually test end to end to graphite). > However, the naming scheme is not ideal for how metric ui's typically would > present the metric tree (e.g. jmx tag syntax doesn't really translate). > > Jason > > On Tue, Jan 27, 2015 at 11:19 AM, Jun Rao <j...@confluent.io> wrote: > > > Jason, Kyle, > > > > I created an 0.8.2 blocker > > https://issues.apache.org/jira/browse/KAFKA-1902 > > and attached a patch there. Could you test it out and see if it fixes the > > issue with the reporter? The patch adds tags as scope in MetricName. > > > > Thanks, > > > > Jun > > > > On Tue, Jan 27, 2015 at 7:39 AM, Jun Rao <j...@confluent.io> wrote: > > > > > Jason, > > > > > > So, this sounds like a real issue. Perhaps we can fix it just by > setting > > > the tag name as the scope. For example, for mbean kafka.server:type= > > > BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have > > > > > > group: "kafka.server" > > > type: "BrokerTopicMetrics" > > > name: "BytesInPerSec" > > > scope: "topic=test" > > > > > > Do you know if scope can have characters like "=" and "," (e.g., for > > scope > > > like "topic=test,partition=1")? > > > > > > The issue with using mytopic-BytesInPerSec as the name is what we are > > > trying to fix in kafka-1481. Topic name (and clientId, etc) can have > dash > > > in it and it's hard to parse. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > > > > On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > >> Remember multiple people have reported this issue. Per topic metrics > no > > >> longer appear in graphite (or in any system modeled after the yammer > > >> GraphiteReporter). They are not being seen as unique. > > >> > > >> While these metrics are registered in the registry as separate > > >> ‘MetricName’ > > >> instances (varying only by mbeanName), the GraphiteReporter sends the > > >> metrics to graphite using only the 4 fields I describe above. > > >> Consequently, > > >> multiple metrics in the registry get sent to graphite under the same > > name. > > >> Thus these metrics all end up in the same bucket in graphite, > trampling > > >> over each other making them useless. They aren’t ‘double counted’ so > > much > > >> as flapping between multiple independent values. > > >> > > >> We actually have our own Reporter class (based off the yammer > > >> GraphiteReporter). Our version sends metrics through kafka which is > then > > >> consumed downstream by multiple metric consumers. > > >> > > >> The ConsoleReporter isn’t useful for actually persisting metrics > > anywhere. > > >> It’s just iterating through all the (identically named metrics in the > > >> registry (save for the different mbeanNames))…. > > >> > > >> The mbeanName, as constructed, is not useful as a human readable > metric > > >> name, to be presented in a browsable tree of metrics, etc. The > > >> ‘group’:’type’:’name’:’scope’ are the pieces that matter. > > >> > > >> The fix here is to produce MetricName instances similar to 0.8.1.1, > etc. > > >> In > > >> this case, it should probably be something like: > > >> > > >> group: kafka.server > > >> type: BrokerTopicMetrics > > >> name: mytopic-BytesInPerSec > > >> group: <unused> > > >> > > >> Jason > > >> > > >> > > >> On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy < > ku...@nmsworks.co.in> > > >> wrote: > > >> > > >> > I have enabled yammer's ConsoleReporter and I am getting all the > > metrics > > >> > (including per-topic metrics). > > >> > > > >> > Yammer's MetricName object implements equals/hashcode methods using > > >> > mBeanName . We are constructing a unique mBeanName for each metric, > So > > >> we > > >> > are not missing/overwriting any metrics. > > >> > > > >> > Current confusion is due to MetricName.name(). This will be same > > >> > (BytesInPerSec) for both broker level and topic level metrics. We > need > > >> to > > >> > use MetricName.getMBeanName() to differentiate between broker level > > and > > >> > topic level metrics. > > >> > > > >> > 0.8.1 MBeanName: > > >> > > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" > > >> > > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec" > > >> > > > >> > 0.8.2 MBeanName: > > >> > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec > > >> > > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC > > >> > > > >> > > > >> > ConsoleReporter's O/P: > > >> > > > >> > BytesInPerSec: <- This is broker level > > >> > count = 1521 > > >> > mean rate = 3.63 bytes/s > > >> > 1-minute rate = 0.35 bytes/s > > >> > 5-minute rate = 2.07 bytes/s > > >> > 15-minute rate = 1.25 bytes/s > > >> > > > >> > BytesInPerSec: <- This is for topic1 > > >> > count = 626 > > >> > mean rate = 1.89 bytes/s > > >> > 1-minute rate = 0.42 bytes/s > > >> > 5-minute rate = 31.53 bytes/s > > >> > 15-minute rate = 64.66 bytes/s > > >> > > > >> > BytesInPerSec: <- This is for topic2 > > >> > count = 895 > > >> > mean rate = 3.62 bytes/s > > >> > 1-minute rate = 1.39 bytes/s > > >> > 5-minute rate = 30.08 bytes/s > > >> > 15-minute rate = 50.27 bytes/s > > >> > > > >> > Manikumar > > >> > > > >> > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <j...@squareup.com> > > >> wrote: > > >> > > > >> > > Ok, > > >> > > > > >> > > It looks like the yammer MetricName is not being created correctly > > for > > >> > the > > >> > > sub metrics that include a topic. E.g. a metric with an mbeanName > > >> like: > > >> > > > > >> > > > > >> > > > >> > > > kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic > > >> > > > > >> > > appears to be malformed. A yammer MetricName has 4 fields that are > > >> used > > >> > in > > >> > > creating a graphite metric, that are included in the constructor: > > >> > > group, type, name, scope. > > >> > > > > >> > > In this case, the metric with the above mbeanName has these fields > > >> set in > > >> > > the MetricName: > > >> > > > > >> > > group: "kafka.server" > > >> > > type: "BrokerTopicMetrics" > > >> > > name: "BytesInPerSec" > > >> > > scope: null > > >> > > > > >> > > Thus, the topic metrics all look the same, and get lumped into the > > >> > > top-level BrokerTopicMetrics (and thus that will now be double > > >> counted). > > >> > It > > >> > > looks like the fix for kafka-1481 was where things got broken. It > > >> seems > > >> > to > > >> > > have introduced ‘tags’ in the building of metric names, and then > > those > > >> > tags > > >> > > only get applied to the mbeanName, but get excluded from the > metric > > >> name: > > >> > > > > >> > > > > >> > > > >> > > > https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f > > >> > > > > >> > > This is a pretty severe issue, since the yammer metrics for these > > >> stats > > >> > > will be double counted in aggregate, and the per-topic stats will > be > > >> > > removed. > > >> > > > > >> > > I should note too, in my previous email, I thought that only the > > >> > per-topic > > >> > > BrokerTopicMetrics were missing, but also several other per-topic > > >> metrics > > >> > > are missing too, e.g. under kafka.log, etc. > > >> > > > > >> > > Jason > > >> > > > > >> > > > > >> > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg < > j...@squareup.com> > > >> > wrote: > > >> > > > > >> > > > I can confirm that the per topic metrics are not coming through > to > > >> the > > >> > > > yammer metrics registry. I do see them in jmx (via jconsole), > but > > >> the > > >> > > > MetricsRegistry does not have them. > > >> > > > All the other metrics are coming through that appear in jmx. > > >> > > > > > >> > > > This is with single node instance running locally. > > >> > > > > > >> > > > Jason > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy < > > >> ku...@nmsworks.co.in > > >> > > > > >> > > > wrote: > > >> > > > > > >> > > >> If you are using multi-node cluster, then metrics may be > reported > > >> from > > >> > > >> other servers. > > >> > > >> pl check all the servers in the cluster. > > >> > > >> > > >> > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker < > > kyleban...@gmail.com > > >> > > > >> > > >> wrote: > > >> > > >> > > >> > > >> > I've been using a custom KafkaMetricsReporter to report Kafka > > >> broker > > >> > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes > and > > >> > > >> messages in > > >> > > >> > and out for all topics together and for each individual > topic. > > >> > > >> > > > >> > > >> > After upgrading to v0.8.2.0, these metrics are no longer > being > > >> > > reported. > > >> > > >> > > > >> > > >> > I'm only seeing the following: > > >> > > >> > BrokerTopicMetrics > > >> > > >> > - BytesInPerSec > > >> > > >> > - BytesOutPerSec > > >> > > >> > - BytesRejectedPerSec > > >> > > >> > - MessagesInPerSec > > >> > > >> > > > >> > > >> > What's more, despite lots of successful writes to the > cluster, > > >> the > > >> > > >> values > > >> > > >> > for these remaining metrics are all zero. > > >> > > >> > > > >> > > >> > I saw that there was some refactoring of metric naming code. > > Was > > >> the > > >> > > >> > behavior supposed to have changed? > > >> > > >> > > > >> > > >> > Many thanks in advance. > > >> > > >> > > > >> > > >> > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
