RE: trouble sending produce requests to 0.9.0.0 broker cluster

[Dana Powers]

Looks like you aren't setting the request client-id, and server is crashing
on it. I'm not sure whether server api is expected to work w/o client-id,
but you can probably fix by sending one. Fwiw, kafka-python sends
'kafka-python' unless user specifies something else.

-Dana
On Jan 11, 2016 8:41 AM,  wrote:

> I forgot to include some information about this problem.  When I sent
> the produce request, the following appeared in server.log:
>
> [2016-01-10 23:03:44,957] ERROR [KafkaApi-3] error when handling
> request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId:
> null; RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition:
> [topic_1,3] -> 37 (kafka.server.KafkaApis)
> java.lang.NullPointerException
>at
> org.apache.kafka.common.metrics.JmxReporter.getMBeanName(JmxReporter.java:127)
>at
> org.apache.kafka.common.metrics.JmxReporter.addAttribute(JmxReporter.java:106)
>at
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)
>at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
>at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
>at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
>at
> kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
>at
> kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
>at
> kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
>at
> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>at
> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>at
> kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
>at
> kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
>at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
>at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>at java.lang.Thread.run(Thread.java:745)
>
> Later when I tried sending another produce request I got a somewhat
> similar error:
>
> [2016-01-11 08:15:05,153] ERROR [KafkaApi-3] error when handling
> request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId:
> null; RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition:
> [topic_1,3] -> 37 (kafka.server.KafkaApis)
> java.lang.IllegalArgumentException: A metric named 'MetricName
> [name=throttle-time, group=Produce, description=Tracking average
> throttle-time per client, tags={client-id=null}]' already exists, can't
> register another one.
>at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:285)
>at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
>at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
>at
> kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
>at
> kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
>at
> kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
>at
> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>at
> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>at
> kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
>at
> kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
>at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
>at
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>at java.lang.Thread.run(Thread.java:745)
>
> In both cases I was running kafka-console-consumer.sh to consume
> messages from the topic I was sending to, and the consumer did see the
> message I sent.
>
> Dave
>
>
> -Original Message-
> From: d...@dspeterson.com
> Sent: Monday, January 11, 2016 12:32am
> To: users@kafka.apache.org
> Subject: trouble sending produce requests to 0.9.0.0 broker cluster
>
> Hi,
>
> I'm having trouble sending produce requests to a Kafka 0.9.0.0 broker
> cluster consisting of 4 brokers with IDs 0, 1, 2, and 3.  All 4
> brokers are running locally on my CentOS 7 development box, listening
> on ports 9092, 9093, 9094, and 9095 respectively.  I am running my
> own producer code (Bruce, see https://github.com/ifwe/bruce), which
> works without problems with Kafka 0.8, but has problems with 0.9.0.0.
> When I send a produce request consisting of a single message, I often
> get a response consisting of error ACK -1 (Unknown, unexpected server
> error) although I have also seen other errors such as 6
> (NotLeaderForPartition).  During one observed instance of this
> behavior I saw the 

RE: trouble sending produce requests to 0.9.0.0 broker cluster

[dave]

I forgot to include some information about this problem.  When I sent
the produce request, the following appeared in server.log:

[2016-01-10 23:03:44,957] ERROR [KafkaApi-3] error when handling request 
Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId: null; 
RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition: [topic_1,3] -> 37 
(kafka.server.KafkaApis)
java.lang.NullPointerException
   at 
org.apache.kafka.common.metrics.JmxReporter.getMBeanName(JmxReporter.java:127)
   at 
org.apache.kafka.common.metrics.JmxReporter.addAttribute(JmxReporter.java:106)
   at 
org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)
   at 
org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
   at 
kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
   at 
kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
   at 
kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
   at 
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
   at 
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
   at 
kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
   at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
   at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
   at java.lang.Thread.run(Thread.java:745)

Later when I tried sending another produce request I got a somewhat
similar error:

[2016-01-11 08:15:05,153] ERROR [KafkaApi-3] error when handling request 
Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId: null; 
RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition: [topic_1,3] -> 37 
(kafka.server.KafkaApis)
java.lang.IllegalArgumentException: A metric named 'MetricName 
[name=throttle-time, group=Produce, description=Tracking average throttle-time 
per client, tags={client-id=null}]' already exists, can't register another one.
   at 
org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:285)
   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
   at 
kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
   at 
kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
   at 
kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
   at 
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
   at 
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
   at 
kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
   at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
   at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
   at java.lang.Thread.run(Thread.java:745)

In both cases I was running kafka-console-consumer.sh to consume
messages from the topic I was sending to, and the consumer did see the
message I sent.

Dave


-Original Message-
From: d...@dspeterson.com
Sent: Monday, January 11, 2016 12:32am
To: users@kafka.apache.org
Subject: trouble sending produce requests to 0.9.0.0 broker cluster

Hi,

I'm having trouble sending produce requests to a Kafka 0.9.0.0 broker
cluster consisting of 4 brokers with IDs 0, 1, 2, and 3.  All 4
brokers are running locally on my CentOS 7 development box, listening
on ports 9092, 9093, 9094, and 9095 respectively.  I am running my
own producer code (Bruce, see https://github.com/ifwe/bruce), which
works without problems with Kafka 0.8, but has problems with 0.9.0.0.
When I send a produce request consisting of a single message, I often
get a response consisting of error ACK -1 (Unknown, unexpected server
error) although I have also seen other errors such as 6
(NotLeaderForPartition).  During one observed instance of this
behavior I saw the following:

sent produce request

- single message set containing a single message
- topic: "topic_1"
- partition: 3
- empty key
- value: "hello world"
- API key: 0 (produce request)
- API version: 0
- correlation ID: 1
- empty client ID
- required ACKs: 1
- timeout: 1
- message magic byte: 0
- message attributes: 0 (no compression)
- destination broker ID: 3

received produce response

Re: trouble sending produce requests to 0.9.0.0 broker cluster

[dave]

I created the following bug report:
https://issues.apache.org/jira/browse/KAFKA-3088


-Original Message-
From: "Dave Peterson" <d...@dspeterson.com>
Sent: Monday, January 11, 2016 9:56am
To: users@kafka.apache.org
Subject: Re: trouble sending produce requests to 0.9.0.0 broker cluster

Ok, thanks for the information.  I tried setting the client ID to 
"bruce" and the
problem disappeared.  I looked at
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
and it doesn't say anything about an empty client ID not being allowed (even
if it isn't allowed, Kafka shouldn't crash on receiving such a request).
Therefore I think this should be considered a bug, and a regression from 
0.8.
As a workaround, I'll avoid sending requests with an empty client ID.

Thanks,
Dave


On 1/11/2016 9:22 AM, Dana Powers wrote:
> Looks like you aren't setting the request client-id, and server is crashing
> on it. I'm not sure whether server api is expected to work w/o client-id,
> but you can probably fix by sending one. Fwiw, kafka-python sends
> 'kafka-python' unless user specifies something else.
>
> -Dana
> On Jan 11, 2016 8:41 AM, <d...@dspeterson.com> wrote:
>
>> I forgot to include some information about this problem.  When I sent
>> the produce request, the following appeared in server.log:
>>
>>  [2016-01-10 23:03:44,957] ERROR [KafkaApi-3] error when handling
>> request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId:
>> null; RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition:
>> [topic_1,3] -> 37 (kafka.server.KafkaApis)
>>  java.lang.NullPointerException
>> at
>> org.apache.kafka.common.metrics.JmxReporter.getMBeanName(JmxReporter.java:127)
>> at
>> org.apache.kafka.common.metrics.JmxReporter.addAttribute(JmxReporter.java:106)
>> at
>> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)
>> at
>> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
>> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
>> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
>> at
>> kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
>> at
>> kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
>> at
>> kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
>> at
>> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>> at
>> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>> at
>> kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
>> at
>> kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
>> at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
>> at
>> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Later when I tried sending another produce request I got a somewhat
>> similar error:
>>
>>  [2016-01-11 08:15:05,153] ERROR [KafkaApi-3] error when handling
>> request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId:
>> null; RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition:
>> [topic_1,3] -> 37 (kafka.server.KafkaApis)
>>  java.lang.IllegalArgumentException: A metric named 'MetricName
>> [name=throttle-time, group=Produce, description=Tracking average
>> throttle-time per client, tags={client-id=null}]' already exists, can't
>> register another one.
>> at
>> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:285)
>> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
>> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
>> at
>> kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
>> at
>> kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
>> at
>> kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
>> at
>> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>> at
>> kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
>> at
>> kafka.server.ReplicaManager.appen

Re: trouble sending produce requests to 0.9.0.0 broker cluster

[Dave Peterson]

Ok, thanks for the information.  I tried setting the client ID to 
"bruce" and the

problem disappeared.  I looked at
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
and it doesn't say anything about an empty client ID not being allowed (even
if it isn't allowed, Kafka shouldn't crash on receiving such a request).
Therefore I think this should be considered a bug, and a regression from 
0.8.

As a workaround, I'll avoid sending requests with an empty client ID.

Thanks,
Dave


On 1/11/2016 9:22 AM, Dana Powers wrote:

Looks like you aren't setting the request client-id, and server is crashing
on it. I'm not sure whether server api is expected to work w/o client-id,
but you can probably fix by sending one. Fwiw, kafka-python sends
'kafka-python' unless user specifies something else.

-Dana
On Jan 11, 2016 8:41 AM,  wrote:


I forgot to include some information about this problem.  When I sent
the produce request, the following appeared in server.log:

 [2016-01-10 23:03:44,957] ERROR [KafkaApi-3] error when handling
request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId:
null; RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition:
[topic_1,3] -> 37 (kafka.server.KafkaApis)
 java.lang.NullPointerException
at
org.apache.kafka.common.metrics.JmxReporter.getMBeanName(JmxReporter.java:127)
at
org.apache.kafka.common.metrics.JmxReporter.addAttribute(JmxReporter.java:106)
at
org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)
at
org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
at
kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
at
kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
at
kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
at
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
at
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
at
kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
at
kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)

Later when I tried sending another produce request I got a somewhat
similar error:

 [2016-01-11 08:15:05,153] ERROR [KafkaApi-3] error when handling
request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId:
null; RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition:
[topic_1,3] -> 37 (kafka.server.KafkaApis)
 java.lang.IllegalArgumentException: A metric named 'MetricName
[name=throttle-time, group=Produce, description=Tracking average
throttle-time per client, tags={client-id=null}]' already exists, can't
register another one.
at
org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:285)
at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
at
kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209)
at
kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111)
at
kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353)
at
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
at
kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371)
at
kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348)
at
kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366)
at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)

In both cases I was running kafka-console-consumer.sh to consume
messages from the topic I was sending to, and the consumer did see the
message I sent.

Dave


-Original Message-
From: d...@dspeterson.com
Sent: Monday, January 11, 2016 12:32am
To: users@kafka.apache.org
Subject: trouble sending produce requests to 0.9.0.0 broker cluster

Hi,

I'm having trouble sending produce requests to a Kafka 0.9.0.0 broker
cluster consisting of 4 brokers with IDs 0, 1, 2, and 3.  All 4
brokers are running locally on my CentOS 7 development box, listening
on ports 9092, 9093, 9094, and 9095 respectively.  I am running my
own producer