[ https://issues.apache.org/jira/browse/KAFKA-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606957#comment-16606957 ]
Ashish Surana edited comment on KAFKA-7149 at 9/7/18 10:34 AM: --------------------------------------------------------------- [~guozhang] As you suggested, below changes can reduce the assignment data size especially when a topic-group has multiple input topics as partitions are repeated for each input-topic in assignment-info in that case: 1. Replace num.partitions with num.tasks 2. Replace [topic-partitions] with [task-ids] This will also require some changes to identify the corresponding host while doing lookup for a given key in store. Along with these encoding changes, compression is definitely required for further size reduction. [~mjsax] suggestion to handle this is what I was also thinking and makes sense to support backward/forward compatibility. was (Author: asurana): [~guozhang] As you suggested, below changes can reduce the assignment data size especially when a topic-group has multiple input topics as topic-partitions are repeated in assignment-info in that case: 1. Replace num.partitions with num.tasks 2. Replace [topic-partitions] with [task-ids] This will also require some changes to identify the corresponding host while doing lookup for a given key in store. Along with these encoding changes, compression is definitely required for further size reduction. [~mjsax] suggestion to handle this is what I was also thinking and makes sense to support backward/forward compatibility. > Reduce assignment data size to improve kafka streams scalability > ---------------------------------------------------------------- > > Key: KAFKA-7149 > URL: https://issues.apache.org/jira/browse/KAFKA-7149 > Project: Kafka > Issue Type: Improvement > Components: streams > Reporter: Ashish Surana > Assignee: Ashish Surana > Priority: Major > > We observed that when we have high number of partitions, instances or > stream-threads, assignment-data size grows too fast and we start getting > below RecordTooLargeException at kafka-broker. > Workaround of this issue is commented at: > https://issues.apache.org/jira/browse/KAFKA-6976 > Still it limits the scalability of kafka streams as moving around 100MBs of > assignment data for each rebalancing affects performance & reliability > (timeout exceptions starts appearing) as well. Also this limits kafka streams > scale even with high max.message.bytes setting as data size increases pretty > quickly with number of partitions, instances or stream-threads. > > Solution: > To address this issue in our cluster, we are sending the compressed > assignment-data. We saw assignment-data size reduced by 8X-10X. This improved > the kafka streams scalability drastically for us and we could now run it with > more than 8,000 partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)