[
https://issues.apache.org/jira/browse/KAFKA-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298866#comment-17298866
]
Tamas Kornai edited comment on KAFKA-12431 at 3/10/21, 2:26 PM:
----------------------------------------------------------------
The issue turned out to be related to how we defined default quotas. We used
the following settings:
{{kafka-configs --zookeeper <zk-address>:2181 --alter --add-config
'producer_byte_rate=0, consumer_byte_rate=0, request_percentage=1000'
--entity-type users --entity-default}}
In version 2.4 the `0` rate was interpreted as enabling the tracking of quota
metrics but no actual limits were enforced. From >=2.5 versions the `0` limit
does trigger rate limiting on clients. Setting more reasonable (i.e. large)
producer/consumer rate values fixed the high CPU issue that we observed.
was (Author: tkornai):
The issue turned out to be related to how we defined default quotas. We used
the following settings:
{{kafka-configs --zookeeper <zk-address>:2181 --alter --add-config
'producer_byte_rate=0, consumer_byte_rate=0, request_percentage=1000'
--entity-type users --entity-default}}
In version 2.4 the `0` rate was interpreted as enabling the tracking of quota
metrics but no actual limits were enforced. From >=2.5 versions the `0` limit
does trigger rate limiting on consumers. Setting more reasonable (i.e. large)
producer/consumer rate values fixed the high CPU issue that we observed.
> Fetch Request/Response without Topic information
> ------------------------------------------------
>
> Key: KAFKA-12431
> URL: https://issues.apache.org/jira/browse/KAFKA-12431
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Peter Sinoros-Szabo
> Priority: Major
> Attachments: fetch-on-2.4.1.png, fetch-on-2.6.1.png,
> kafka-highcpu-24.svg.zip, kafka-highcpu-26.svg.zip
>
>
> I was running a 6 node Kafka 2.4.1 cluster with protocol and message format
> version set to 2.4. I wanted to upgrade the cluster to 2.6.1 and after I
> upgraded the 1st broker to 2.6.1 without any configuration change, I noticed
> much higher CPU usage on that broker (instead of 25% CPU usage it was ~350%)
> and about 3-4x higher network traffic. So I dumped the traffic between the
> Kafka client and broker and compared it with the traffic of the same broker
> downgraded to 2.4.1.
> It seems to me that after I upgraded to 2.6.1, the Fetch requests and
> responses are not complete, it is missing the topics part of the Fetch
> Request, I don't know for what reason. I guess there should be always a
> topics part.
> I'll attache a screenshot from these messages.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
