[
https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504977#comment-16504977
]
rajadayalan perumalsamy edited comment on KAFKA-7012 at 6/7/18 6:31 PM:
------------------------------------------------------------------------
Thanks, my comments added inline
the attached profile screenshot shows only 210ms (out of >650k ms) of self time
in pool.tryAllocate().
a possible time drain with the patch is determineHandlingOrder() - it may
shuffle the sockets in order to guarantee fairness under low memory conditions
- but it doesnt appear in the screenshot.
- 'did not notice issue with memory, i can do profiling with memory and
include it.'
there's also no equivalent screenshot of a detailed profile breakdown for the
results without the patch, so no base to compare to.']
- attaching screenshot Commit-f15cdbc91b-profile2 for comparing.
was the profiler using cpu time or wall clock time for measurements?
- 'In the attached screenshot Total time is the wall clock time and Total
time(CPU) is the cpu time.'
were the ssl settings (cipher used etc) the same across measurements?
- 'ssl settings and cipher used are same across measurements.'
was (Author: rajadayalanvdms):
Thanks, my comments added inline
the attached profile screenshot shows only 210ms (out of >650k ms) of self time
in pool.tryAllocate().
a possible time drain with the patch is determineHandlingOrder() - it may
shuffle the sockets in order to guarantee fairness under low memory conditions
- but it doesnt appear in the screenshot.
- 'did not notice issue with memory, i can do profiling with memory and
include it.
there's also no equivalent screenshot of a detailed profile breakdown for the
results without the patch, so no base to compare to.'
- attaching screenshot Commit-f15cdbc91b-profile2 for comparing.
was the profiler using cpu time or wall clock time for measurements?
- 'In the attached screenshot Total time is the wall clock time and Total
time(CPU) is the cpu time.'
were the ssl settings (cipher used etc) the same across measurements?
- 'ssl settings and cipher used are same across measurements.'
> Performance issue upgrading to kafka 1.0.1 or 1.1
> -------------------------------------------------
>
> Key: KAFKA-7012
> URL: https://issues.apache.org/jira/browse/KAFKA-7012
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 1.1.0, 1.0.1
> Reporter: rajadayalan perumalsamy
> Assignee: praveen
> Priority: Major
> Attachments: Commit-47ee8e954-profile.png,
> Commit-47ee8e954-profile2.png, Commit-f15cdbc91b-profile.png,
> Commit-f15cdbc91b-profile2.png
>
>
> We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1.
> After upgrading 1 node on the cluster, we notice that network threads use
> most of the cpu. It is a 3 node cluster with 15k messages/sec on each node.
> With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60%
> vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is
> high depending on the number of network threads used. If networks threads is
> set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4
> then the cpu usage is around 450%(5 vcpus). Using the same kafka
> server.properties for both.
> Did further analysis with git bisect, couple of build and deploys, traced the
> issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine
> for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit
> 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have
> attached screenshots of profiling done with both the commits. Screenshot
> Commit-f15cdbc91b-profile shows less cpu usage by network threads and
> Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show
> higher cpu usage(almost entire cpu usage) by network threads. Also noticed
> that kafka.network.Processor.poll() method is invoked 10 times more with
> commit 47ee8e954df62b9a79099e944ec4be29afe046f6.
> We need the issue to be resolved to upgrade the cluster. Please let me know
> if you need any additional information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
