[
https://issues.apache.org/jira/browse/KAFKA-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946118#comment-15946118
]
Charly Molter commented on KAFKA-3992:
--------------------------------------
This is not a contribution.
While I understand this limit it does sound to me that asking for unique
client-id per thread seems like an unreasonable ask for the following reason:
1) Client-ids is a concept that survives all the way to the broker. Having to
do aggregation this means there would be an explosion in the number of metrics
for the brokers
2) Security and Quotas are heavily depending on client-id for .e.g this is
extracted from the docs "can be applied to (user, client-id), user or client-id
groups”. Adding 1000 entries to quota to allow an app with a 1000threads might
be a bit annoying
3) The doc is clearly not saying it should be unique: "An id string to pass to
the server when making requests. The purpose of this is to be able to track the
source of requests beyond just ip/port by allowing a logical application name
to be included in server-side request logging.” or "The client id is a
user-specified string sent in each request to help trace calls. It should
logically identify the application making the request.”
4) KIP-98 does introduce producer-id which is supposed to be unique if
client-id is already unique what’s the point of this producer-id?
It doesn’t seem that client id was created to identify a specific instance but
to identify an application (which may have multiple instances of clients).
So It seems it’s either unclear in the docs or a problem in the metrics API.
What do you think [~ewencp]?
> InstanceAlreadyExistsException Error for Consumers Starting in Parallel
> -----------------------------------------------------------------------
>
> Key: KAFKA-3992
> URL: https://issues.apache.org/jira/browse/KAFKA-3992
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0, 0.10.0.0
> Reporter: Alexander Cook
> Assignee: Ewen Cheslack-Postava
>
> I see the following error sometimes when I start multiple consumers at about
> the same time in the same process (separate threads). Everything seems to
> work fine afterwards, so should this not actually be an ERROR level message,
> or could there be something going wrong that I don't see?
> Let me know if I can provide any more info!
> Error processing messages: Error registering mbean
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> org.apache.kafka.common.KafkaException: Error registering mbean
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
>
> Caused by: javax.management.InstanceAlreadyExistsException:
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> Here is the full stack trace:
> M[?:com.ibm.streamsx.messaging.kafka.KafkaConsumerV9.produceTuples:-1] -
> Error processing messages: Error registering mbean
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> org.apache.kafka.common.KafkaException: Error registering mbean
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> at
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:159)
> at
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:77)
> at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
> at
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:641)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:268)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:126)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorKnown(AbstractCoordinator.java:186)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:857)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829)
> at
> com.ibm.streamsx.messaging.kafka.KafkaConsumerV9.produceTuples(KafkaConsumerV9.java:129)
> at
> com.ibm.streamsx.messaging.kafka.KafkaConsumerV9$1.run(KafkaConsumerV9.java:70)
> at java.lang.Thread.run(Thread.java:785)
> at
> com.ibm.streams.operator.internal.runtime.OperatorThreadFactory$2.run(OperatorThreadFactory.java:137)
> Caused by: javax.management.InstanceAlreadyExistsException:
> kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=node--1
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:449)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1910)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:978)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:912)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:336)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:534)
> at
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:157)
> ... 18 more
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
