Hi, everyone, We are fixing the mbean names in kafka-1482, by adding separate explicit tags in the name for things like clientId and topic. Another thing that some people have complained before is that we use quotes in the jmx name. Should we also just get rid of the quotes as part of kafka-1482? So, instead of "kafka.server":type="BrokerTopicMetrics",name="topic-1-BytesInPerSec" we will have kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1
Thanks, Jun On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > I am going to vote for 1482 to be included in 0.8.2, if we have a patch > submitted in a week. I think we've had this JIRA opened for too long and we > held people back so it's only fair to release this. > > On Wed, Oct 8, 2014 at 9:40 PM, Jun Rao <jun...@gmail.com> wrote: > > > Otis, > > > > Just have the patch ready asap. We can make a call then. > > > > Thanks, > > > > Jun > > > > On Wed, Oct 8, 2014 at 6:13 AM, Otis Gospodnetic < > > otis.gospodne...@gmail.com > > > wrote: > > > > > Hi Jun, > > > > > > Would by the end of next week be acceptable for 0.8.2? > > > > > > Thanks, > > > Otis > > > -- > > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > On Tue, Oct 7, 2014 at 4:04 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > Otis, > > > > > > > > Yes, if you guys can help provide a patch in a few days, we can > > probably > > > > get it to the 0.8.2 release. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic < > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > Hi Jun, > > > > > > > > > > I think your MBean renaming approach will work. I see > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version > > > 0.8.2, > > > > > but > > > > > is not marked as a Blocker. We'd love to get the MBeans fixed so > > this > > > > > makes it in 0.8.2 release. Do you know if this is on anyone's > plate > > > (the > > > > > issue is currently Unassigned)? If not, should we provide a new > > patch > > > > that > > > > > uses your approach? > > > > > > > > > > Thanks, > > > > > Otis > > > > > -- > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > Management > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > > > > > Otis, > > > > > > > > > > > > In kafka-1481, we will have to change the mbean names (at least > the > > > > ones > > > > > > with clientid and topic) anyway. Using the name/value pair in the > > > mbean > > > > > > name allows us to do this in a cleaner way. Yes, "," is not > allowed > > > in > > > > > > clientid or topic. > > > > > > > > > > > > Bhavesh, > > > > > > > > > > > > Yes, I was thinking of making changes in the new metrics package. > > > > > Something > > > > > > like allowing the sensor names to have name/value pairs. The jmx > > > names > > > > > will > > > > > > just follow accordingly. This is probably cleaner than doing the > > > > > escaping. > > > > > > Also, the metric names are more intuitive (otherwise, you have to > > > know > > > > > > which part is the clientid and which part is the topic). > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic < > > > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <jun...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > Bhavesh, > > > > > > > > > > > > > > > > Yes, allowing dot in clientId and topic makes it a bit harder > > to > > > > > define > > > > > > > the > > > > > > > > JMX bean names. I see a couple of solutions here. > > > > > > > > > > > > > > > > 1. Disable dot in clientId and topic names. The issue is that > > dot > > > > may > > > > > > > > already be used in existing deployment. > > > > > > > > > > > > > > > > 2. We can represent the JMX bean name differently in the new > > > > > producer. > > > > > > > > Instead of > > > > > > > > kafka.producer.myclientid:type=mytopic > > > > > > > > we could change it to > > > > > > > > kafka.producer:clientId=myclientid,topic=mytopic > > > > > > > > > > > > > > > > I felt that option 2 is probably better since it doesn't > affect > > > > > > existing > > > > > > > > users. > > > > > > > > > > > > > > > > > > > > > > If it doesn't affect existing users, great. > > > > > > > > > > > > > > If you are saying that each "piece" of MBean name could be > > > expressed > > > > as > > > > > > > name=value pair, with something like "," (forbidden in host > > names, > > > > > topic > > > > > > > names, client IDs, etc. I assume?) then yes, I think this would > > be > > > > > easier > > > > > > > to parse and it would be easier for people to understand what > is > > > > what. > > > > > > > > > > > > > > Otis > > > > > > > -- > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > > > Management > > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Otis, > > > > > > > > > > > > > > > > We probably can also use option 2 to address KAFKA-1481. For > > > > > > > topic/clientid > > > > > > > > specific metrics, we could explicitly specify the metric name > > so > > > > that > > > > > > it > > > > > > > > contains "topic=mytopic,clientid=myclientid". That seems to > be > > a > > > > much > > > > > > > > cleaner way than having all parts included in a single string > > > > > separated > > > > > > > by > > > > > > > > '|'. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry < > > > > > > > > mistry.p.bhav...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > HI Otis, > > > > > > > > > > > > > > > > > > What is migration path ? If topic with special chars > exists > > > > > already( > > > > > > > > > ".","-","|" etc) in previous version of producer/consumer > of > > > > > Kafka, > > > > > > > what > > > > > > > > > happens after the upgrade new producer or consumer (kafka > > > > version) > > > > > ? > > > > > > > > Also, > > > > > > > > > in new producer API (Kafka Trunk), does this enforce the > rule > > > > about > > > > > > > > client > > > > > > > > > id as well ? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Bhavesh > > > > > > > > > > > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic < > > > > > > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > So maybe I should I should have asked the Q explicitly: > > > > > > > > > > Could we commit the patch from > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now > > that, I > > > > > hope, > > > > > > > > it's > > > > > > > > > > clear what problems the current MBean names can cause? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Otis > > > > > > > > > > -- > > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized > Log > > > > > > > Management > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic < > > > > > > > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > *Problem:* > > > > > > > > > > > Some Kafka 0.8.x MBeans have names composed of things > > like > > > > > > > <consumer > > > > > > > > > > > group>-<topic>-<metric name>. Note how dashes are used > > as > > > > > > > > delimiters. > > > > > > > > > > > When <consumer group> and <topic> don't contain the > > > > delimiter > > > > > > > > > character > > > > > > > > > > > all is good if you want to extract parts of this MBean > > name > > > > by > > > > > > > simply > > > > > > > > > > > splitting on the delimiter character. The problem is > > that > > > > > dashes > > > > > > > are > > > > > > > > > > > allowed in topic and group names, so this splitting > > doesn't > > > > > work. > > > > > > > > > > > Moreover, underscores are also used as delimiters, and > > they > > > > can > > > > > > > also > > > > > > > > be > > > > > > > > > > > used in things like topic names. > > > > > > > > > > > > > > > > > > > > > > *Example*: > > > > > > > > > > > This MBean's name is composed of <consumer > > > > > > > > group>-<topic>-BytesPerSec: > > > > > > > > > > > > > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > > > > > name="*myGroup**-myTopic**-* > > > > > > > > > > > BytesPerSec" > > > > > > > > > > > > > > > > > > > > > > Here we can actually split on "-" and extract all 3 > parts > > > > from > > > > > > the > > > > > > > > > MBean > > > > > > > > > > > name:: > > > > > > > > > > > * consumer group ('*myGroup*') > > > > > > > > > > > * topic ('*myTopic*') > > > > > > > > > > > * metric (‘BytesPerSec’) > > > > > > > > > > > > > > > > > > > > > > All good! > > > > > > > > > > > > > > > > > > > > > > But imagine if I named the group: *my-Group* > > > > > > > > > > > And if I named the topic: *my-Topic* > > > > > > > > > > > > > > > > > > > > > > Then we'd have: > > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > > > > > > name="*my-Group**-my-Topic**-* > > > > > > > > > > > BytesPerSec" > > > > > > > > > > > > > > > > > > > > > > Now splitting on "-" would no longer work! To extract > > > > > "my-Group" > > > > > > > and > > > > > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know > > the > > > > > > > specific > > > > > > > > > > group > > > > > > > > > > > name and topic name to look for and could not use > generic > > > > > > approach > > > > > > > of > > > > > > > > > > just > > > > > > > > > > > splitting the MBean name on the delimiter. > > > > > > > > > > > > > > > > > > > > > > *Solution*: > > > > > > > > > > > The patch in > > > > https://issues.apache.org/jira/browse/KAFKA-1481 > > > > > > > > replaces > > > > > > > > > > > all _ and - characters where they are used as > delimiters > > in > > > > > MBean > > > > > > > > names > > > > > > > > > > > with a "|" character. Because the "I" character is not > > > > allowed > > > > > > in > > > > > > > > > topic > > > > > > > > > > > names, consumer groups, host names, splitting on this > new > > > and > > > > > > > unified > > > > > > > > > > > delimiter works. > > > > > > > > > > > > > > > > > > > > > > I hope this explains the problem, the solution, and > that > > > this > > > > > can > > > > > > > > make > > > > > > > > > it > > > > > > > > > > > in the next 0.8.x. > > > > > > > > > > > > > > > > > > > > > > Otis > > > > > > > > > > > -- > > > > > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized > > Log > > > > > > > > Management > > > > > > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
