[
https://issues.apache.org/jira/browse/KAFKA-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Jorgensen updated KAFKA-4141:
------------------------------------
Description:
We are seeing about a 2x increase in CPU usage for the new kafka producer
compared to the 0.8.0.1 producer. We are currently using gzip compression.
We recently upgraded our kafka server and producer to 0.10.0.1 from 0.8.1.1 and
noticed that the cpu usage for the new producer had increased pretty
significantly compared to the old producer. This has caused us to need more
resources to do the same amount of work we were doing before. I did some quick
profiling and it looks like during sending half of the cpu cycles are sped in
org.apache.kafka.common.record.Compressor.putRecord and the other half is in
org.apache.kafka.common.record.Record.computeChecksum (both are around 5.8% of
the cpu cycles for that method). I know its not apples to apples but the old
producer did not seem to have this overhead or at least it was greatly reduced.
Is this a known performance degradation compared to the old producer?
h2. old producer:
!http://imgur.com/1xS34Dl.jpg!
h2. new producer:
!http://imgur.com/0w0G5b1.jpg!
was:
We are seeing about a 2x increase in CPU usage for the new kafka producer
compared to the 0.8.0.1 producer. We are currently using gzip compression.
We recently upgraded our kafka server and producer to 0.10.0.1 from 0.8.1.1 and
noticed that the cpu usage for the new producer had increased pretty
significantly compared to the old producer. This has caused us to need more
resources to do the same amount of work we were doing before. I did some quick
profiling and it looks like during sending half of the cpu cycles are sped in
org.apache.kafka.common.record.Compressor.putRecord and the other half is in
org.apache.kafka.common.record.Record.computeChecksum (both are around 5.8% of
the cpu cycles for that method). I know its not apples to apples but the old
producer did not seem to have this overhead or at least it was greatly reduced.
Is this a known performance degradation compared to the old producer?
Here is the trace from the old producer:
!http://imgur.com/1xS34Dl.jpg!
New producer:
!http://imgur.com/0w0G5b1.jpg!
> 2x increase in cpu usage on new Producer API
> --------------------------------------------
>
> Key: KAFKA-4141
> URL: https://issues.apache.org/jira/browse/KAFKA-4141
> Project: Kafka
> Issue Type: Bug
> Reporter: Andrew Jorgensen
>
> We are seeing about a 2x increase in CPU usage for the new kafka producer
> compared to the 0.8.0.1 producer. We are currently using gzip compression.
> We recently upgraded our kafka server and producer to 0.10.0.1 from 0.8.1.1
> and noticed that the cpu usage for the new producer had increased pretty
> significantly compared to the old producer. This has caused us to need more
> resources to do the same amount of work we were doing before. I did some
> quick profiling and it looks like during sending half of the cpu cycles are
> sped in org.apache.kafka.common.record.Compressor.putRecord and the other
> half is in org.apache.kafka.common.record.Record.computeChecksum (both are
> around 5.8% of the cpu cycles for that method). I know its not apples to
> apples but the old producer did not seem to have this overhead or at least it
> was greatly reduced.
> Is this a known performance degradation compared to the old producer?
> h2. old producer:
> !http://imgur.com/1xS34Dl.jpg!
> h2. new producer:
> !http://imgur.com/0w0G5b1.jpg!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
