Jonathan Gordon created KAFKA-7652:
--------------------------------------
Summary: Kafka Streams Session store performance degradation from
0.10.2.2 to 0.11.0.0
Key: KAFKA-7652
URL: https://issues.apache.org/jira/browse/KAFKA-7652
Project: Kafka
Issue Type: Bug
Affects Versions: 2.0.1, 2.0.0, 1.1.1, 0.11.0.3, 0.11.0.2, 0.11.0.1,
0.11.0.0
Reporter: Jonathan Gordon
Attachments: kafka_10_2_1_flushes.txt, kafka_11_0_3_flushes.txt
Here's the original thread from the mailing list:
https://lists.apache.org/thread.html/97d620f4fd76be070ca4e2c70e2fda53cafe051e8fc4505dbcca0321@%3Cusers.kafka.apache.org%3E
We are attempting to upgrade our Kafka Streams application from 0.10.2.1 but
experience a severe performance degradation. The highest amount of CPU time
seems spent in retrieving from the local cache. Here's an example with 0.11.0.0:
[https://i.imgur.com/l5VEsC2.png]
When things are running smoothly we're gated by retrieving from the state store
with acceptable performance. Here's an example with 0.10.2.1:
[https://i.imgur.com/IHxC2cZ.png]
Some investigation reveals that it appears we're performing about 3 orders
magnitude more lookups on the NamedCache over a comparable time period. I've
attached logs of the NamedCache flush logs for 0.10.2.1 and 0.11.0.3.
We're using session windows and have the app configured for commit.interval.ms
of 30 * 1000 and cache.max.bytes.buffering = 10485760
I'm happy to share more details if they would be helpful. Also happy to run
tests on our data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
