Ashish Surana created KAFKA-7149:
------------------------------------
Summary: Reduce assignment data size to improve kafka streams
scalability
Key: KAFKA-7149
URL: https://issues.apache.org/jira/browse/KAFKA-7149
Project: Kafka
Issue Type: Improvement
Reporter: Ashish Surana
We observed that when we have high number of partitions, instances or
stream-threads, assignment-data size grows too fast and we start getting below
exception at kafka-broker.
RecordTooLargeException
Resolution of this issue is explained at:
https://issues.apache.org/jira/browse/KAFKA-6976
Still it limits the scalability of kafka streams as moving around 100MBs of
assignment data for each rebalancing affects performance & reliability (timeout
exceptions starts appearing) as well. Also this limits kafka streams scale even
with high max.message.bytes setting as data size increases pretty quickly with
number of partitions, instances or stream-threads.
Solution:
To address this issue in our cluster, we are sending the compressed
assignment-data. We saw assignment-data size reduced by 8X-10X. This improved
the kafka streams scalability drastically for us and we could now run it with
more than 8,000 partitions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
