[
https://issues.apache.org/jira/browse/KAFKA-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322246#comment-14322246
]
Jun Rao commented on KAFKA-1952:
--------------------------------
Uploaded a new patch that addresses the review comments.
Some test results.
Create a 2 node cluster locally. Create a topic X with 3000 partitions and a
replication factor of 2 (wait until all leaders are elected; this can take 5
minutes). Without the patch, when there is no produce load on the cluster, each
broker uses 100% CPU and the localTime for each replica fetch request is about
1 sec. With the patch, each broker uses about 10% CPU and the localTime for
each replica fetch request is about 9ms.
Also tested the end to end latency with the patch. On the above 2 node cluster,
create a topic test with 1 partition and a replication factor of 2.
bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency localhost:9092
localhost:2181 test 5000 500 -1
0 61.206
1000 26.766
2000 38.58
3000 25.647
4000 26.991
Avg latency: 28.5952 ms
Percentiles: 50th = 27, 99th = 41, 99.9th = 52
The latency is bounded by the message commit time. With 3000 partitions, each
replica fetch request take 9ms in localTime and another 9ms in sendTime. So the
time to complete a produce request (with ack=-1) is about 26ms on average. In
any case, the end to end latency is way less than the maxWaitTime in the
regular consumer and replica fetcher (both at 500ms), which is expected.
Also measured the localTime and sendTime for the same replica fetch requests on
0.8.1.1. The numbers are comparable with 0.8.2.0 with the patch.
> High CPU Usage in 0.8.2 release
> -------------------------------
>
> Key: KAFKA-1952
> URL: https://issues.apache.org/jira/browse/KAFKA-1952
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2.0
> Reporter: Jay Kreps
> Assignee: Jun Rao
> Priority: Critical
> Fix For: 0.8.2.0
>
> Attachments: kafka-1952.patch, kafka-1952_2015-02-15_15:26:33.patch
>
>
> Brokers with high partition count see increased CPU usage when migrating from
> 0.8.1.1 to 0.8.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
