[jira] [Created] (KAFKA-1877) Expose version via JMX for 'new' producer
Vladimir Tretyakov created KAFKA-1877:
-
Summary: Expose version via JMX for 'new' producer
Key: KAFKA-1877
URL: https://issues.apache.org/jira/browse/KAFKA-1877
Project: Kafka
Issue Type: Bug
Components: clients, producer
Affects Versions: 0.8.2
Reporter: Vladimir Tretyakov
Fix For: 0.8.3
Add version of Kafka to jmx (monitoring tool can use this info).
Something like that
{code}
kafka.common:type=AppInfo,name=Version
Value java.lang.Object = 0.8.2-beta
{code}
we already have this in core Kafka module (see kafka.common.AppInfo object).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276944#comment-14276944
]
Vladimir Tretyakov commented on KAFKA-1723:
---
Hi, sorry for delay, but maybe possible to add version of Kafka to jmx
(monitoring tool can use this info).
Something like that
{code}
kafka.common:type=AppInfo,name=Version
Value java.lang.Object = 0.8.2-beta
{code}
we already have this in core Kafka module (see kafka.common.AppInfo object).
Thx and sorry for delay.
make the metrics name in new producer more standard
---
Key: KAFKA-1723
URL: https://issues.apache.org/jira/browse/KAFKA-1723
Project: Kafka
Issue Type: Sub-task
Components: clients
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Manikumar Reddy
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch,
KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch,
KAFKA-1723_2015-01-09_23:43:22.patch, KAFKA-1723_2015-01-14_13:30:39.patch
The jmx name in the new producer looks like the following:
kafka.producer.myclientid:type=mytopic
However, this can be ambiguous since we allow . in client id and topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1723) make the metrics name in new producer more standard
[
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267397#comment-14267397
]
Vladimir Tretyakov edited comment on KAFKA-1723 at 1/7/15 8:25 AM:
---
Hi, thx for patch, one question from me, in patch I see something like that:
{code}
+MapString, String tags = new LinkedHashMapString, String();
+tags.put(clientId, clientId);
+tags.put(topic, topic);
+
-topicRecordCount.add(topic. + topic + .record-send-rate,
new Rate());
+topicRecordCount.add(topic. + topic + .record-send-rate,
ProducerTopicMetrics, tags, new Rate());
{code}
Note that (topic. + topic + .record-send-rate part and tags both contain
topic.
Can we avoid duplication and use in all places something like:
{code}
topicRecordCount.add(record-send-rate, ProducerTopicMetrics, tags, new
Rate());
{code}
?
Thx.
was (Author: vladimir.tretyakov):
Hi, thx for patch, one question from me, in patch I see something like that:
{code}
+MapString, String tags = new LinkedHashMapString, String();
+tags.put(clientId, clientId);
+tags.put(topic, topic);
+
-topicRecordCount.add(topic. + topic + .record-send-rate,
new Rate());
+topicRecordCount.add(topic. + topic + .record-send-rate,
ProducerTopicMetrics, tags, new Rate());
{code}
Note that (topic. + topic + .record-send-rate part and tags both contain
topic.
Can we avoid duplication and use in all places something like:
topicRecordCount.add(record-send-rate, ProducerTopicMetrics, tags, new
Rate());
?
Thx.
make the metrics name in new producer more standard
---
Key: KAFKA-1723
URL: https://issues.apache.org/jira/browse/KAFKA-1723
Project: Kafka
Issue Type: Sub-task
Components: clients
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Manikumar Reddy
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1723.patch
The jmx name in the new producer looks like the following:
kafka.producer.myclientid:type=mytopic
However, this can be ambiguous since we allow . in client id and topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: logFlushes.png Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Assignee: Vladimir Tretyakov Priority: Critical Labels: patch Fix For: 0.8.2, 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, logFlushes.png, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258109#comment-14258109
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi Jun, thx for answer, it helps (log.flush.interval.messages property), I
see data on charts
!https://issues.apache.org/jira/secure/attachment/12689014/logFlushes.png!
Maybe you also have a receipt how to force displaying of these metrics
kafka.log.LogCleaner:
{code}
newGauge(max-buffer-utilization-percent,
new Gauge[Int] {
def value: Int = cleaners.map(_.lastStats).map(100 *
_.bufferUtilization).max.toInt
})
/* a metric to track the recopy rate of each thread's last cleaning */
newGauge(cleaner-recopy-percent,
new Gauge[Int] {
def value: Int = {
val stats = cleaners.map(_.lastStats)
val recopyRate = stats.map(_.bytesWritten).sum.toDouble /
math.max(stats.map(_.bytesRead).sum, 1)
(100 * recopyRate).toInt
}
})
/* a metric to track the maximum cleaning time for the last cleaning from
each thread */
newGauge(max-clean-time-secs,
new Gauge[Int] {
def value: Int =
cleaners.map(_.lastStats).map(_.elapsedSecs).max.toInt
})
{code}
?
Thx again.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
Labels: patch
Fix For: 0.8.2, 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch,
KAFKA-1481_2014-11-14_16-39-41_doc.patch,
KAFKA-1481_2014-11-17_14-33-03.patch,
KAFKA-1481_2014-11-19_16-03-03_trunk.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, logFlushes.png, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: (was: logFlushes.png) Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Assignee: Vladimir Tretyakov Priority: Critical Labels: patch Fix For: 0.8.2, 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, logflushes.png, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: logflushes.png Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Assignee: Vladimir Tretyakov Priority: Critical Labels: patch Fix For: 0.8.2, 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, logflushes.png, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258109#comment-14258109
]
Vladimir Tretyakov edited comment on KAFKA-1481 at 12/24/14 8:44 AM:
-
Hi Jun, thx for answer, it helps (log.flush.interval.messages property), I
see data on charts
!https://issues.apache.org/jira/secure/attachment/12689019/logflushes.png!
Maybe you also have a receipt how to force displaying of these metrics
kafka.log.LogCleaner:
{code}
newGauge(max-buffer-utilization-percent,
new Gauge[Int] {
def value: Int = cleaners.map(_.lastStats).map(100 *
_.bufferUtilization).max.toInt
})
/* a metric to track the recopy rate of each thread's last cleaning */
newGauge(cleaner-recopy-percent,
new Gauge[Int] {
def value: Int = {
val stats = cleaners.map(_.lastStats)
val recopyRate = stats.map(_.bytesWritten).sum.toDouble /
math.max(stats.map(_.bytesRead).sum, 1)
(100 * recopyRate).toInt
}
})
/* a metric to track the maximum cleaning time for the last cleaning from
each thread */
newGauge(max-clean-time-secs,
new Gauge[Int] {
def value: Int =
cleaners.map(_.lastStats).map(_.elapsedSecs).max.toInt
})
{code}
?
Thx again.
was (Author: vladimir.tretyakov):
Hi Jun, thx for answer, it helps (log.flush.interval.messages property), I
see data on charts
!https://issues.apache.org/jira/secure/attachment/12689014/logFlushes.png!
Maybe you also have a receipt how to force displaying of these metrics
kafka.log.LogCleaner:
{code}
newGauge(max-buffer-utilization-percent,
new Gauge[Int] {
def value: Int = cleaners.map(_.lastStats).map(100 *
_.bufferUtilization).max.toInt
})
/* a metric to track the recopy rate of each thread's last cleaning */
newGauge(cleaner-recopy-percent,
new Gauge[Int] {
def value: Int = {
val stats = cleaners.map(_.lastStats)
val recopyRate = stats.map(_.bytesWritten).sum.toDouble /
math.max(stats.map(_.bytesRead).sum, 1)
(100 * recopyRate).toInt
}
})
/* a metric to track the maximum cleaning time for the last cleaning from
each thread */
newGauge(max-clean-time-secs,
new Gauge[Int] {
def value: Int =
cleaners.map(_.lastStats).map(_.elapsedSecs).max.toInt
})
{code}
?
Thx again.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
Labels: patch
Fix For: 0.8.2, 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch,
KAFKA-1481_2014-11-14_16-39-41_doc.patch,
KAFKA-1481_2014-11-17_14-33-03.patch,
KAFKA-1481_2014-11-19_16-03-03_trunk.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, logflushes.png, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255722#comment-14255722
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi guys, maybe it is not a best place for this question, but it is related to
metric, so I will try:
In code ( kafka/core/src/main/scala/kafka/log/FileMessageSet.scala ) I see:
{code}
object LogFlushStats extends KafkaMetricsGroup {
val logFlushTimer = new KafkaTimer(newTimer(LogFlushRateAndTimeMs,
TimeUnit.MILLISECONDS, TimeUnit.SECONDS))
}
{code}
But I don't see this metric in JMX when I attach JConsole to my broker.
Is it ok?
What I have to do to see this metric?
Maybe this bean/metric exists short time and recreates each time after/during
log flushes?
Can somebody explain me?
Best regards. Vladimir.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
Labels: patch
Fix For: 0.8.2, 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch,
KAFKA-1481_2014-11-14_16-39-41_doc.patch,
KAFKA-1481_2014-11-17_14-33-03.patch,
KAFKA-1481_2014-11-19_16-03-03_trunk.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255875#comment-14255875 ] Vladimir Tretyakov commented on KAFKA-1481: --- Already did, waited more than 1 day (on our test env), nothing, no bean with such name is in broker process. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Assignee: Vladimir Tretyakov Priority: Critical Labels: patch Fix For: 0.8.2, 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255875#comment-14255875 ] Vladimir Tretyakov edited comment on KAFKA-1481 at 12/22/14 4:07 PM: - Hi Jun, thx for answer, already did (added many messages), waited more than 1 day (on our test env), nothing, no bean with such name is in broker process. was (Author: vladimir.tretyakov): Already did, waited more than 1 day (on our test env), nothing, no bean with such name is in broker process. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Assignee: Vladimir Tretyakov Priority: Critical Labels: patch Fix For: 0.8.2, 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-11-19_16-03-03_trunk.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-11-17_14-33-03.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214559#comment-14214559 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, provided new patch, last one?:) 80.1 Done 80.2 Done 80.3 Done 80.4 Do you mean 'new producer'? Can I just reformulate KAFKA-1768? Something like: Expose version via JMX in new client? 81. Done 82. Done 83. Left as is. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214559#comment-14214559 ] Vladimir Tretyakov edited comment on KAFKA-1481 at 11/17/14 11:52 AM: -- Hi, provided new patch, last one?:) 80.1 Done 80.2 Done 80.3 Done 80.4 Do you mean 'new producer'? Can I just reformulate KAFKA-1768? Something like: Expose version via JMX in new client? 81. Done 82. Done 83. Left as is. PS: I'll be traveling for 3 week starting 19.11 (Wednesday) and won't Internet access. Can we get this committed before I go pleaase? was (Author: vladimir.tretyakov): Hi, provided new patch, last one?:) 80.1 Done 80.2 Done 80.3 Done 80.4 Do you mean 'new producer'? Can I just reformulate KAFKA-1768? Something like: Expose version via JMX in new client? 81. Done 82. Done 83. Left as is. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-11-14_16-39-41_doc.patch KAFKA-1481_2014-11-14_16-33-03.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212264#comment-14212264 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, added new patches (code + doc), go by way 3: (3) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec Also added Kafka version MBean, it exposes only Kafka version now (from gradle.properties file). I didn't find easy way where I can get build hash, so only version for now. I hope it will be my last patches, it is a time consumption to change things many times and test everything each time and prepare patched, so I really hope these patches are good enough and I will not do additional iterations, thx. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212598#comment-14212598 ] Vladimir Tretyakov commented on KAFKA-1481: --- Thx Jun, will try to fix everything according your last comments. re 83, yeah host:port is unique pair, so it will work even with KAFKA-1684 Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208284#comment-14208284 ] Vladimir Tretyakov commented on KAFKA-1481: --- Maybe somebody can answer my last questions? Have to finish with this patch and moving forward! Thx. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208284#comment-14208284
]
Vladimir Tretyakov edited comment on KAFKA-1481 at 11/12/14 5:54 PM:
-
Maybe somebody can answer my last questions? Have to finish with this patch and
moving forward! Thx.
Will extract Kafka version like Gwen Shapira
has suggested in
http://search-hadoop.com/m/4TaT4xtk36/Programmatic+Kafka+version+detection%252Fextractionsubj=Programmatic+Kafka+version+detection+extraction+
{quote}
So it looks like we can use Gradle to add properties to manifest file and
then use getResourceAsStream to read the file and parse it.
The Gradle part would be something like:
jar.manifest {
attributes('Implementation-Title': project.name,
'Implementation-Version': project.version,
'Built-By': System.getProperty('user.name'),
'Built-JDK': System.getProperty('java.version'),
'Built-Host': getHostname(),
'Source-Compatibility': project.sourceCompatibility,
'Target-Compatibility': project.targetCompatibility
)
}
The code part would be:
this.getClass().getClassLoader().getResourceAsStream(/META-INF/MANIFEST.MF)
Does that look like the right approach?
{quote}
What do you think?
What about 65?
{quote}
{quote}
was (Author: vladimir.tretyakov):
Maybe somebody can answer my last questions? Have to finish with this patch and
moving forward! Thx.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206303#comment-14206303
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi [~junrao],
{quote}
Vladimir Tretyakov, we could probably just add a new MBean to expose the Kafka
version number. Any value in exposing other things like build hash and build
timestamp?
{quote}
Version will be ok I think, but my question was also about where I can get this
Version? Should I hardcoded this Version in MBean? What is the best place for
such information? Maybe you already have some special property file?
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206333#comment-14206333
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi again,
65. Honestly I'd prefer allBrokers=true and allTopics=true or maybe
brokers=aggregated, topics=aggregated.
From first view user will not know what this is about:
{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964
{code}
and only after user will see something like:
{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964,
broker=0
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964,
broker=1
{code}
user can understand (not sure) that like without 'broker=..' is aggregated
value, I'd prefer explicit definition.
[~jjkoshy] what do you think about:
brokers=aggregated, topics=aggregated ? In jconsole user will see
aggregated instead of true, handy from my point of view.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204467#comment-14204467
]
Vladimir Tretyakov edited comment on KAFKA-1481 at 11/10/14 10:07 AM:
--
Hi [~jjkoshy] thx for your feedback, will change, one question from me:
What if I include Kafka version in any of the MBeans? It will easy for
external tools to detect Kafka version and use parsing rules for particular
version. Something like Elasticsearch provides by http (Kafka will provide by
JMX).
{code}
curl localhost:9200
{
...
version : {
number : 1.3.2,
build_hash : dee175dbe2f254f3f26992f5d7591939aaefd12f,
build_timestamp : 2014-08-13T14:29:30Z
}
}
{code}
Maybe you can point out code place where I can get current Kafka version?
was (Author: vladimir.tretyakov):
Hi [~jjkoshy] thx for your feedback, will change, one question from me:
What if I include Kafka version in any of the MBeans? It will easy for
external tools to detect Kafka version and use parsing rules for particular
version. Something like Elasticsearch provides by http (Kafka will provide by
JMX).
{code}
curl localhost:9200
{
...
version : {
number : 1.3.2,
build_hash : dee175dbe2f254f3f26992f5d7591939aaefd12f,
build_timestamp : 2014-08-13T14:29:30Z
}
}
{code}
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-11-10_20-39-41_doc.patch KAFKA-1481_2014-11-10_21-02-23.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204467#comment-14204467
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi [~jjkoshy] thx for your feedback, will change, one question from me:
What if I include Kafka version in any of the MBeans? It will easy for
external tools to detect Kafka version and use parsing rules for particular
version. Something like Elasticsearch provides by http (Kafka will provide by
JMX).
{code}
curl localhost:9200
{
...
version : {
number : 1.3.2,
build_hash : dee175dbe2f254f3f26992f5d7591939aaefd12f,
build_timestamp : 2014-08-13T14:29:30Z
}
}
{code}
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1100) metrics shouldn't have generation/timestamp specific names
[
https://issues.apache.org/jira/browse/KAFKA-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198433#comment-14198433
]
Vladimir Tretyakov commented on KAFKA-1100:
---
Right [~otis], names will be:
{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,allBrokers=true
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,brokerHost=wawanawna,brokerPort=9092
{code}
metrics shouldn't have generation/timestamp specific names
--
Key: KAFKA-1100
URL: https://issues.apache.org/jira/browse/KAFKA-1100
Project: Kafka
Issue Type: Bug
Reporter: Jason Rosenberg
I've noticed that there are several metrics that seem useful for monitoring
overtime, but which contain generational timestamps in the metric name.
We are using yammer metrics libraries to send metrics data in a background
thread every 10 seconds (to kafka actually), and then they eventually end up
in a metrics database (graphite, opentsdb). The metrics then get graphed via
UI, and we can see metrics going way back, etc.
Unfortunately, many of the metrics coming from kafka seem to have metric
names that change any time the server or consumer is restarted, which makes
it hard to easily create graphs over long periods of time (spanning app
restarts).
For example:
names like:
kafka.consumer.FetchRequestAndResponseMetricssquare-1371718712833-e9bb4d10-0-508818741-AllBrokersFetchRequestRateAndTimeMs
or:
kafka.consumer.ZookeeperConsumerConnector...topicName.square-1373476779391-78aa2e83-0-FetchQueueSize
In our staging environment, we have our servers on regular auto-deploy cycles
(they restart every few hours). So just not longitudinally usable to have
metric names constantly changing like this.
Is there something that can easily be done? Is it really necessary to have
so much cryptic info in the metric name?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1100) metrics shouldn't have generation/timestamp specific names
[
https://issues.apache.org/jira/browse/KAFKA-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198433#comment-14198433
]
Vladimir Tretyakov edited comment on KAFKA-1100 at 11/5/14 2:35 PM:
Right [~otis], names will be:
{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,allBrokers=true
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,brokerHost=wawanawna,brokerPort=9092
kafka.consumer:type=ZookeeperConsumerConnector,name=FetchQueueSize,clientId=af_servers,topic=spm_topic,threadId=0
{code}
was (Author: vladimir.tretyakov):
Right [~otis], names will be:
{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,allBrokers=true
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,brokerHost=wawanawna,brokerPort=9092
{code}
metrics shouldn't have generation/timestamp specific names
--
Key: KAFKA-1100
URL: https://issues.apache.org/jira/browse/KAFKA-1100
Project: Kafka
Issue Type: Bug
Reporter: Jason Rosenberg
I've noticed that there are several metrics that seem useful for monitoring
overtime, but which contain generational timestamps in the metric name.
We are using yammer metrics libraries to send metrics data in a background
thread every 10 seconds (to kafka actually), and then they eventually end up
in a metrics database (graphite, opentsdb). The metrics then get graphed via
UI, and we can see metrics going way back, etc.
Unfortunately, many of the metrics coming from kafka seem to have metric
names that change any time the server or consumer is restarted, which makes
it hard to easily create graphs over long periods of time (spanning app
restarts).
For example:
names like:
kafka.consumer.FetchRequestAndResponseMetricssquare-1371718712833-e9bb4d10-0-508818741-AllBrokersFetchRequestRateAndTimeMs
or:
kafka.consumer.ZookeeperConsumerConnector...topicName.square-1373476779391-78aa2e83-0-FetchQueueSize
In our staging environment, we have our servers on regular auto-deploy cycles
(they restart every few hours). So just not longitudinally usable to have
metric names constantly changing like this.
Is there something that can easily be done? Is it really necessary to have
so much cryptic info in the metric name?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198439#comment-14198439
]
Vladimir Tretyakov commented on KAFKA-1723:
---
Hi [~junrao],
{quote}
It uses a proprietary newly developed metrics package.
{quote}
Does that mean that users who will use new producer will see 'old' style
metric's name?
{quote}
proprietary newly developed metrics package
{quote}
Maybe it is possible to add new metrics in sync with
https://issues.apache.org/jira/browse/KAFKA-1481 ?
Just worry that in future Kafka may have many metric's naming conventions,
sorry:)
make the metrics name in new producer more standard
---
Key: KAFKA-1723
URL: https://issues.apache.org/jira/browse/KAFKA-1723
Project: Kafka
Issue Type: Sub-task
Components: clients
Affects Versions: 0.8.2
Reporter: Jun Rao
Fix For: 0.8.3
The jmx name in the new producer looks like the following:
kafka.producer.myclientid:type=mytopic
However, this can be ambiguous since we allow . in client id and topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198580#comment-14198580 ] Vladimir Tretyakov commented on KAFKA-1723: --- Good news, thx [~junrao] make the metrics name in new producer more standard --- Key: KAFKA-1723 URL: https://issues.apache.org/jira/browse/KAFKA-1723 Project: Kafka Issue Type: Sub-task Components: clients Affects Versions: 0.8.2 Reporter: Jun Rao Fix For: 0.8.3 The jmx name in the new producer looks like the following: kafka.producer.myclientid:type=mytopic However, this can be ambiguous since we allow . in client id and topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-11-03_16-39-41_doc.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-11-03_17-02-23.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194551#comment-14194551 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, added new patch 50. I think it is better to have separate case class for 'all' things. It is real 'other' case. 51. Done 52.1 Done 52.2 Done 52.3 Done 52.4 Done 52.5 Done 53. https://issues.apache.org/jira/secure/attachment/12678927/KAFKA-1481_2014-11-03_16-39-41_doc.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-31_14-35-43.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191698#comment-14191698 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi Jun, added new one, changed according to your last comments. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-30_21-35-43.patch.patch.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-24_14-14-35.patch.patch, KAFKA-1481_2014-10-30_21-35-43.patch.patch.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14190581#comment-14190581 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi Jun, added new patch, removed Taggable, did everything as you have suggested. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-24_14-14-35.patch.patch, KAFKA-1481_2014-10-30_21-35-43.patch.patch.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: (was: KAFKA-1481_2014-10-24_14-14-35.patch.patch) Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-30_21-35-43.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188553#comment-14188553
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi Jun, thx for juicy feedback again, one question:
{quote}
31. AbstractFetcherThread:
31.1 You changed the meaning of clientId. clientId is used in the fetch request
and we want to leave it as just the clientId string. Since the clientId should
be uniquely representing a particular consumer client, we just need to include
the clientId in the metric name. We don't need to include the consumer id in
either the fetch request or the metric name since it's too long and has
redundant info.
{quote}
I didn't change meaning of clientId here, look (all code without my changes):
consumerIdString string is:
{code}
val consumerIdString = {
var consumerUuid : String = null
config.consumerId match {
case Some(consumerId) // for testing only
= consumerUuid = consumerId
case None // generate unique consumerId automatically
= val uuid = UUID.randomUUID()
consumerUuid = %s-%d-%s.format(
InetAddress.getLocalHost.getHostName, System.currentTimeMillis,
uuid.getMostSignificantBits().toHexString.substring(0,8))
}
config.groupId + _ + consumerUuid
}
{code}
thread name is (consumerIdString + fetcherId + sourceBroker.id):
{code}
override def createFetcherThread(fetcherId: Int, sourceBroker: Broker):
AbstractFetcherThread = {
new ConsumerFetcherThread(
ConsumerFetcherThread-%s-%d-%d.format(consumerIdString, fetcherId,
sourceBroker.id),
config, sourceBroker, partitionMap, this)
}
{code}
clientId inside AbstractFetcherThread is: config.clientId + consumerIdString +
fetcherId + sourceBroker.id
{code}
class ConsumerFetcherThread(name: String,
val config: ConsumerConfig,
sourceBroker: Broker,
partitionMap: Map[TopicAndPartition,
PartitionTopicInfo],
val consumerFetcherManager: ConsumerFetcherManager)
extends AbstractFetcherThread(name = name,
clientId = config.clientId + - + name,
sourceBroker = sourceBroker,
socketTimeout = config.socketTimeoutMs,
socketBufferSize =
config.socketReceiveBufferBytes,
fetchSize = config.fetchMessageMaxBytes,
fetcherBrokerId =
Request.OrdinaryConsumerId,
maxWait = config.fetchWaitMaxMs,
minBytes = config.fetchMinBytes,
isInterruptible = true) {
{code}
As you see there is no clean clientId inside AbstractFetcherThread class and it
is not unique situation, and this is main goal why I added Taggable.
Now I am trying to remove Taggable, but I have no idea what to do with such
cases I've described. Can I add new 'clientId' parameter to all these classes
and use only this new/clean clientId as part of matric name? Any other
suggestions? Thx.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch,
KAFKA-1481_2014-10-24_14-14-35.patch.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188553#comment-14188553
]
Vladimir Tretyakov edited comment on KAFKA-1481 at 10/29/14 4:49 PM:
-
Hi Jun, thx for juicy feedback again, one question:
{quote}
31. AbstractFetcherThread:
31.1 You changed the meaning of clientId. clientId is used in the fetch request
and we want to leave it as just the clientId string. Since the clientId should
be uniquely representing a particular consumer client, we just need to include
the clientId in the metric name. We don't need to include the consumer id in
either the fetch request or the metric name since it's too long and has
redundant info.
{quote}
I didn't change meaning of clientId here, look (all code without my changes):
consumerIdString string is:
{code}
val consumerIdString = {
var consumerUuid : String = null
config.consumerId match {
case Some(consumerId) // for testing only
= consumerUuid = consumerId
case None // generate unique consumerId automatically
= val uuid = UUID.randomUUID()
consumerUuid = %s-%d-%s.format(
InetAddress.getLocalHost.getHostName, System.currentTimeMillis,
uuid.getMostSignificantBits().toHexString.substring(0,8))
}
config.groupId + _ + consumerUuid
}
{code}
thread name is (consumerIdString + fetcherId + sourceBroker.id):
{code}
override def createFetcherThread(fetcherId: Int, sourceBroker: Broker):
AbstractFetcherThread = {
new ConsumerFetcherThread(
ConsumerFetcherThread-%s-%d-%d.format(consumerIdString, fetcherId,
sourceBroker.id),
config, sourceBroker, partitionMap, this)
}
{code}
clientId inside AbstractFetcherThread is: config.clientId + consumerIdString +
fetcherId + sourceBroker.id
{code}
class ConsumerFetcherThread(name: String,
val config: ConsumerConfig,
sourceBroker: Broker,
partitionMap: Map[TopicAndPartition,
PartitionTopicInfo],
val consumerFetcherManager: ConsumerFetcherManager)
extends AbstractFetcherThread(name = name,
clientId = config.clientId + - + name,
sourceBroker = sourceBroker,
socketTimeout = config.socketTimeoutMs,
socketBufferSize =
config.socketReceiveBufferBytes,
fetchSize = config.fetchMessageMaxBytes,
fetcherBrokerId =
Request.OrdinaryConsumerId,
maxWait = config.fetchWaitMaxMs,
minBytes = config.fetchMinBytes,
isInterruptible = true) {
{code}
As you see there is no clean clientId inside AbstractFetcherThread class and it
is not unique situation, and this is main goal why I added Taggable.
Now I am trying to remove Taggable, but I have no idea what to do with such
cases I've described. Can I add new 'clientId' parameter to all these classes
and use only this new/clean clientId as part of matric name? Any other
suggestions? Thx.
PS: there is no way t get 'clean' parameters in such classes and build map for
metrics just here, I must stretching parameters during all classes as Taggable.
PSS: Can we use more interactive way for communication, Jira is not a bets way
for discussion/explanation, not fast at least:)
was (Author: vladimir.tretyakov):
Hi Jun, thx for juicy feedback again, one question:
{quote}
31. AbstractFetcherThread:
31.1 You changed the meaning of clientId. clientId is used in the fetch request
and we want to leave it as just the clientId string. Since the clientId should
be uniquely representing a particular consumer client, we just need to include
the clientId in the metric name. We don't need to include the consumer id in
either the fetch request or the metric name since it's too long and has
redundant info.
{quote}
I didn't change meaning of clientId here, look (all code without my changes):
consumerIdString string is:
{code}
val consumerIdString = {
var consumerUuid : String = null
config.consumerId match {
case Some(consumerId) // for testing only
= consumerUuid = consumerId
case None // generate unique consumerId automatically
= val uuid = UUID.randomUUID()
consumerUuid = %s-%d-%s.format(
InetAddress.getLocalHost.getHostName, System.currentTimeMillis,
uuid.getMostSignificantBits().toHexString.substring(0,8))
}
config.groupId + _ + consumerUuid
}
{code}
thread name is (consumerIdString + fetcherId + sourceBroker.id):
{code}
override def createFetcherThread(fetcherId: Int, sourceBroker: Broker):
AbstractFetcherThread = {
new
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-24_14-14-35.patch.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-24_14-14-35.patch.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182670#comment-14182670
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi Jun, thx for your feedback again, attached new patch.
20.1 done.
20.2 clientId is Taggable here, it will convert to something like
clientId=XXX automatically
21 tags should be Map which iterators iterate in the same order elements were
inserted (final string must be stable): from LinkedHashMap docs The iterator
and all traversal methods of this class visit elements in the order they were
inserted.. I am new in scala, if you know better candidate with predictable
iterators please let me know
22.1 done
22.2 clientId is not always just clientId=XXX, look at:
{code}
class ConsumerFetcherThread(consumerFetcherThreadId: ConsumerFetcherThreadId,
val config: ConsumerConfig,
sourceBroker: Broker,
partitionMap: Map[TopicAndPartition,
PartitionTopicInfo],
val consumerFetcherManager: ConsumerFetcherManager)
extends AbstractFetcherThread(name = consumerFetcherThreadId,
clientId = new TaggableInfo(new
TaggableInfo(clientId - config.clientId), consumerFetcherThreadId),
sourceBroker = sourceBroker,
socketTimeout = config.socketTimeoutMs,
socketBufferSize =
config.socketReceiveBufferBytes,
fetchSize = config.fetchMessageMaxBytes,
fetcherBrokerId =
Request.OrdinaryConsumerId,
maxWait = config.fetchWaitMaxMs,
minBytes = config.fetchMinBytes,
isInterruptible = true) {
{code}
this clientId = new TaggableInfo(new TaggableInfo(clientId -
config.clientId), consumerFetcherThreadId) part was in code before my changes
(was manipulation with strings, not Taggable of course). Yeah, somewhere in
code we can use string instead of Taggable, but maybe it is better to has
Taggable everywhere what is related to metrics. At least it was not easy to
me decide where I have to use Taggable, but where I can leave string. I'd
prefer Taggable everywhere in this case, sorry:).
23. done everything as described by link, checked how it works/applies on
'origin/0.8.2' from scratch.
24. see 22.2
25. agree, added second constructor with 'String' as clientId, can't remove
constructor with Taggable as clientId because as you can see in 22.2
sometimes clientId is not just clientId but something more complex.
26. added 'MetricsLeakTest', basic idea is start/stop many producers/consumes
and observe count of metrics in Metrics.defaultRegistry(), count should not
grow.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch,
KAFKA-1481_2014-10-24_14-14-35.patch.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-21_09-14-35.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178039#comment-14178039
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi, added last patch (KAFKA-1481_2014-10-21_09-14-35.patch). Worked for me
locally (previous worked too:( ).
There are commands I've used:
{code}
PATCH CREATION
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git status
On branch 0.8.2
Your branch is up-to-date with 'origin/0.8.2'.
Changes to be committed:
(use git reset HEAD file... to unstage)
new file: core/src/main/scala/kafka/common/BrokerInfo.scala
new file: core/src/main/scala/kafka/common/ClientIdTopic.scala
new file: core/src/main/scala/kafka/common/Taggable.scala
renamed:core/src/main/scala/kafka/common/ClientIdAndTopic.scala -
core/src/main/scala/kafka/common/TopicInfo.scala
new file:
core/src/main/scala/kafka/consumer/ConsumerFetcherThreadId.scala
new file: core/src/main/scala/kafka/consumer/ConsumerId.scala
Changes not staged for commit:
(use git add file... to update what will be committed)
(use git checkout -- file... to discard changes in working directory)
modified: core/src/main/scala/kafka/admin/AdminUtils.scala
modified:
core/src/main/scala/kafka/api/HeartbeatRequestAndHeader.scala
modified:
core/src/main/scala/kafka/api/HeartbeatResponseAndHeader.scala
modified:
core/src/main/scala/kafka/api/JoinGroupRequestAndHeader.scala
modified:
core/src/main/scala/kafka/api/JoinGroupResponseAndHeader.scala
modified: core/src/main/scala/kafka/api/RequestKeys.scala
modified: core/src/main/scala/kafka/cluster/Partition.scala
...
...
...
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git commit -a -m
'kafka-1481; JMX renaming'
[0.8.2 a232af6] kafka-1481; JMX renaming
73 files changed, 666 insertions(+), 435 deletions(-)
create mode 100644 core/src/main/scala/kafka/common/BrokerInfo.scala
create mode 100644 core/src/main/scala/kafka/common/ClientIdTopic.scala
create mode 100644 core/src/main/scala/kafka/common/Taggable.scala
rename core/src/main/scala/kafka/common/{ClientIdAndTopic.scala =
TopicInfo.scala} (68%)
create mode 100644
core/src/main/scala/kafka/consumer/ConsumerFetcherThreadId.scala
create mode 100644 core/src/main/scala/kafka/consumer/ConsumerId.scala
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git log | head -n 20
commit a232af63b8a1f43054994a74520b31ef7b9b347c
Author: wawanawna sema...@mail.com
Date: Tue Oct 21 09:29:31 2014 +0300
kafka-1481; JMX renaming
commit eb7ac9eb7eebc4e0655b65e07cae594e61a6c05e
Author: Jun Rao jun...@gmail.com
Date: Mon Oct 20 11:09:31 2014 -0700
kafka-1717; remove netty dependency through ZK 3.4.x; patched by Jun Rao;
reviewed by Sriharsha Chintalapani and Neha Narkhede
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git diff
eb7ac9eb7eebc4e0655b65e07cae594e61a6c05e
a232af63b8a1f43054994a74520b31ef7b9b347c
/tmp/KAFKA-1481_2014-10-21_09-14-35.patch
PATCH APPLYING
wawanawna@wawanawna$ git clone https://github.com/apache/kafka.git
wawanawna@wawanawna:/home/storage/sematext/src/kfk3/kafka$ git checkout -b
0.8.2 origin/0.8.2
Branch 0.8.2 set up to track remote branch 0.8.2 from origin.
Switched to a new branch '0.8.2'
wawanawna@wawanawna:/home/storage/sematext/src/kfk3/kafka$ git apply
/tmp/KAFKA-1481_2014-10-21_09-14-35.patch
/tmp/KAFKA-1481_2014-10-21_09-14-35.patch:1410: trailing whitespace.
})
warning: 1 line adds whitespace errors.
{code}
There is warning, but everything works (compilation/tests passed without errors)
PS: please tell me what I am doing wrong.
wawanawna@wawanawna:/home/storage/sematext/src/kfk3/kafka$ git --version
git version 1.9.1
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.2
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178815#comment-14178815 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, Jun, will be perfect! Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177149#comment-14177149
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi, fuf:), created new patch (for 0.8.2 branch).
re 2.1, 2.3 - didn't change this part because we use this part in our
monitoring, it is more handy for user to see something like 'localhost:9092'
instead of '2 (where 2 is broker id)'. For us it is easiest to see/have this
information directly in mBean.
With this patch mBeanNames look like:
{code}
kafka.producer:type=ProducerTopicMetrics,name=MessagesPerSec,topic=spm_alerts_topic
kafka.producer:type=ProducerTopicMetrics,name=MessagesPerSec,allTopics=true
kafka.log:type=Log,name=LogEndOffset,topic=spm_alerts_topic,partitionId=0
kafka.consumer:type=ZookeeperConsumerConnector,name=FetchQueueSize,timestamp=1413817796508,clientId=af_servers,uuid=9f99df40,groupId=af_servers,topicThreadId=spm_topic,consumerHostName=wawanawna,threadId=0
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,threadName=ConsumerFetcherThread,fetcherId=0,sourceBrokerId=0,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413817796508,uuid=9f99df40,brokerHost=wawanawna,brokerPort=9092
{code}
There is not everything clear for me in code, so I've tried to avoid deep
refactoring. All tests passed for me locally.
PS: note that if 'value' part is empty (in key-value structure) we will not
see this part in attributes.
example: key1-value1,key2-null,key3-value3 will be converted to
key1=value1,key3=value3.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.2
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177423#comment-14177423 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, thx for quick reply, did rebase, added new patches (1 was created by git, 1 by IDEA IDE). Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-20_23-14-35.patch KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173425#comment-14173425 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi Jun, thx for juicy feedback, few comments fro me: 1-2. Did it in this way because I didn't want to change existing code a lot, tried got what we need based on current method signatures. I've thought about your approach with Map too, reject it because of larger changes. Now I see that I will do it. 3. Removed 'mBeanName' param, was legacy code. 4. As I've understood from code this method must be called before shutdown in hook, right? We have to remove only metrics related to particular clientId and we shouldn't touch others, right? Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-15_10-23-35.patch KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172118#comment-14172118 ] Vladimir Tretyakov commented on KAFKA-1481: --- Added 2 patches (created with git diff and from IDEA IDE) with a small refactoring in KafkaMetricsGroup. Jun can you please double check and change part related to metrics removing (KAFKA-1567) I can't easily test this code locally, thx a lot. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172118#comment-14172118 ] Vladimir Tretyakov edited comment on KAFKA-1481 at 10/15/14 8:01 AM: - Added 2 patches (created with git diff and from IDEA IDE) with a small refactoring in KafkaMetricsGroup. Jun can you please double check and change part related to metrics removing (KAFKA-1567) I can't easily test this code locally, thx a lot. PS: all tests passed after this patch for me locally. was (Author: vladimir.tretyakov): Added 2 patches (created with git diff and from IDEA IDE) with a small refactoring in KafkaMetricsGroup. Jun can you please double check and change part related to metrics removing (KAFKA-1567) I can't easily test this code locally, thx a lot. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171371#comment-14171371
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Hi, Jun,
Thx that you found a time to look at patch, I've added another regarding your
suggestion, now in JMX you will see (hope it is what you need):
{code}
kafka.server:type=FetcherStats,name=RequestsPerSec,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,topic=spm_topic,partitionId=0
kafka.consumer:type=ZookeeperConsumerConnector,name=OwnedPartitionsCount,clientId=af_servers,groupId=af_servers,topic=spm_topic
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,allBrokers=true
{code}
{quote}
Also, your patch doesn't seem to apply to latest trunk.
git apply ~/Downloads/KAFKA-1481_2014-10-13_18-23-35.patch
error: core/src/main/scala/kafka/common/ClientIdTopic.scala: No such file or
directory
{quote}
Interesting...
1. I've worked with 0.8.2 branch, not trunk
2. ClientIdTopic.scala - I've added this file (must be in patch).
Added 2 equal patches (created them in different way):
1. 'git diff'
2. With help from IDEA IDE.
Try please, maybe one of them will work
Thx again.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.2
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch KAFKA-1481_2014-10-14_21-53-35.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171371#comment-14171371
]
Vladimir Tretyakov edited comment on KAFKA-1481 at 10/14/14 7:09 PM:
-
Hi, Jun,
Thx that you found a time to look at patch, I've added another regarding your
suggestion, now in JMX you will see (hope it is what you need):
{code}
kafka.server:type=FetcherStats,name=RequestsPerSec,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,topic=spm_topic,partitionId=0
kafka.consumer:type=ZookeeperConsumerConnector,name=OwnedPartitionsCount,clientId=af_servers,groupId=af_servers,topic=spm_topic
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,allBrokers=true
{code}
{quote}
Also, your patch doesn't seem to apply to latest trunk.
git apply ~/Downloads/KAFKA-1481_2014-10-13_18-23-35.patch
error: core/src/main/scala/kafka/common/ClientIdTopic.scala: No such file or
directory
{quote}
Interesting...
1. I've worked with 0.8.2 branch, not trunk
2. ClientIdTopic.scala - I've added this file (must be in patch).
Added 2 equal patches (created them in different way):
1. 'git diff' (KAFKA-1481_2014-10-14_21-53-35.patch)
2. With help from IDEA IDE. (KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch)
Try please, maybe one of them will work
Thx again.
was (Author: vladimir.tretyakov):
Hi, Jun,
Thx that you found a time to look at patch, I've added another regarding your
suggestion, now in JMX you will see (hope it is what you need):
{code}
kafka.server:type=FetcherStats,name=RequestsPerSec,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,topic=spm_topic,partitionId=0
kafka.consumer:type=ZookeeperConsumerConnector,name=OwnedPartitionsCount,clientId=af_servers,groupId=af_servers,topic=spm_topic
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,allBrokers=true
{code}
{quote}
Also, your patch doesn't seem to apply to latest trunk.
git apply ~/Downloads/KAFKA-1481_2014-10-13_18-23-35.patch
error: core/src/main/scala/kafka/common/ClientIdTopic.scala: No such file or
directory
{quote}
Interesting...
1. I've worked with 0.8.2 branch, not trunk
2. ClientIdTopic.scala - I've added this file (must be in patch).
Added 2 equal patches (created them in different way):
1. 'git diff'
2. With help from IDEA IDE.
Try please, maybe one of them will work
Thx again.
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.2
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169396#comment-14169396
]
Vladimir Tretyakov commented on KAFKA-1481:
---
Added another patch which introduce metrics name like pairs (key=value),
examples:
{code}
kafka.consumer:type=ConsumerTopicMetrics,name=clientId=af_servers,AllTopics,BytesPerSec
kafka.cluster:type=Partition,name=topic=spm_alerts_topic,partitionId=0,UnderReplicated
kafka.network:type=SocketServer,name=NetworkProcessorNum=1,IdlePercent
kafka.server:type=BrokerTopicMetrics,name=topic=spm_topic,MessagesInPerSec
kafka.server:type=FetcherStats,name=clientId=af_servers,ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413199781501,uuid=58a5cc70,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,BytesPerSec
kafka.server:type=FetcherLagMetrics,name=clientId=af_servers,ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413199781501,uuid=58a5cc70,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,topic=spm_topic,partitionId=0,ConsumerLag
kafka.consumer:type=FetchRequestAndResponseMetrics,name=clientId=af_servers,ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413199781501,uuid=58a5cc70,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,FetchResponseSize
{code}
Stop using dashes AND underscores as separators in MBean names
--
Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.2
Attachments: KAFKA-1481_2014-06-06_13-06-35.patch
MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-13_18-23-35.patch Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Priority: Critical Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093822#comment-14093822 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi Jun Rao, I understood what you suggest, thx, but I am afraid it will not work everywhere or I've missed something. Look at example from my local env: 1) kafka.producer:type=ProducerRequestMetrics,name=af-servers-AllBrokersProducerRequestRateAndTimeMs 2) kafka.consumer:type=ZookeeperConsumerConnector,name=af-servers-af-servers-spm-new-cluster-topic-af-servers-wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize From 1 I can parse af-servers as clientId, ok. From 2 I have to parse: topic ,groupId, consumer host, timestamp, uuid, num_streams. If I remove af-servers from 2 I will get spm-new-cluster-topic-af-servers-wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize. I can extract timestamp, uuid, num_streams without problems, now let me talk about this part spm-new-cluster-topic-af-servers-wawanawna-Dell I see af-servers in the middle but can I be sure that clientId will be always here? Why initial bean name contains af-servers 3 times? I have a feeling sometimes all these 3 parts may be different and I will not able to parse everything in right way. It is painful in general as you mentioned above because we have to remember context for parse each line, I am pretty sure nobody like it. Even if somebody parse names in way you have described I think they will glad to change its parse mechanism with more easiest. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092910#comment-14092910 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi Jun Rao, look at http://search-hadoop.com/m/4TaT4lonIW Example: kafka.consumer:type=ZookeeperConsumerConnector,name=af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize Look at part: spm_new_cluster_topic-af_servers_wawanawna-Dell, what is host name here servers_wawanawna-Dell or wawanawna-Dell ? Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Labels: patch Fix For: 0.8.2 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.2#6252)
