[
https://issues.apache.org/jira/browse/KAFKA-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271902#comment-15271902
]
ASF GitHub Bot commented on KAFKA-3494:
---------------------------------------
GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/1323
KAFKA-3494: add metric id to client mbeans
KafkaConsumer and KafkaProducer mbeans currently only have a client-id
granularity. Client-id represents a logical name for an application.
Given that quotas encourage reuse of client-ids, it's not unexpected to
have multiple clients share the same client-id within the same jvm. When a
later client gets instantiated with the same client-id as an earlier client in
the same jvm, JmxReporter unregisters the original client's mbeans that collide
with the new client's mbeans.
This commit makes client mbeans have a metric-id granularity to prevent
mbean collision and the original client mbean unregistration.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-3494
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1323.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1323
----
commit 40a0189e856ba42be396bd2694e5719e8e29631b
Author: Onur Karaman <[email protected]>
Date: 2016-05-04T23:34:28Z
add metric id to client mbeans
KafkaConsumer and KafkaProducer mbeans currently only have a client-id
granularity. Client-id represents a logical name for an application.
Given that quotas encourage reuse of client-ids, it's not unexpected to
have multiple clients share the same client-id within the same jvm. When a
later client gets instantiated with the same client-id as an earlier client in
the same jvm, JmxReporter unregisters the original client's mbeans that collide
with the new client's mbeans.
This commit makes client mbeans have a metric-id granularity to prevent
mbean collision and the original client mbean unregistration.
----
> mbeans overwritten with identical clients on a single jvm
> ---------------------------------------------------------
>
> Key: KAFKA-3494
> URL: https://issues.apache.org/jira/browse/KAFKA-3494
> Project: Kafka
> Issue Type: Bug
> Reporter: Onur Karaman
>
> Quotas today are implemented on a (client-id, broker) granularity. I think
> one of the motivating factors for using a simple quota id was to allow for
> flexibility in the granularity of the quota enforcement. For instance, entire
> services can share the same id to get some form of (service, broker)
> granularity quotas. From my understanding, client-id was chosen as the quota
> id because it's a property that already exists on the clients and reusing it
> had relatively low impact.
> Continuing the above example, let's say a service spins up multiple
> KafkaConsumers in one jvm sharing the same client-id because they want those
> consumers to be quotad as a single entity. Sharing client-ids within a single
> jvm would cause problems in client metrics since the mbeans tags only go as
> granular as the client-id.
> An easy example is kafka-metrics count. Here's a sample code snippet:
> {code}
> package org.apache.kafka.clients.consumer;
> import java.util.Collections;
> import java.util.Properties;
> import org.apache.kafka.common.TopicPartition;
> public class KafkaConsumerMetrics {
> public static void main(String[] args) throws InterruptedException {
> Properties properties = new Properties();
> properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9092");
> properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test");
> properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
> "org.apache.kafka.common.serialization.StringDeserializer");
>
> properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> "org.apache.kafka.common.serialization.StringDeserializer");
> properties.setProperty(ConsumerConfig.CLIENT_ID_CONFIG,
> "testclientid");
> KafkaConsumer<String, String> kc1 = new KafkaConsumer<>(properties);
> KafkaConsumer<String, String> kc2 = new KafkaConsumer<>(properties);
> kc1.assign(Collections.singletonList(new TopicPartition("t1", 0)));
> while (true) {
> kc1.poll(1000);
> System.out.println("kc1 metrics: " + kc1.metrics().size());
> System.out.println("kc2 metrics: " + kc2.metrics().size());
> Thread.sleep(1000);
> }
> }
> }
> {code}
> jconsole shows one mbean
> kafka.consumer:type=kafka-metrics-count,client-id=testclientid consistently
> with value 40.
> but stdout shows:
> {code}
> kc1 metrics: 69
> kc2 metrics: 40
> {code}
> I think the two possible solutions are:
> 1. add finer granularity to the mbeans to distinguish between the clients
> 2. require client ids to be unique per jvm like KafkaStreams has done
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
