[ https://issues.apache.org/jira/browse/KAFKA-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539596#comment-16539596 ]
Ashish Surana edited comment on KAFKA-7149 at 7/11/18 6:34 AM: --------------------------------------------------------------- Made the change here: [https://github.com/a-surana/kafka/commit/577992015d3bfc5a23e23b5bf32e40a3f92bc74a] Scenario#1: This is straightforward, and works with this. Encoded version: 4 Decoder support latest version: 4 Scenario#2: Encoded version: <=3 (encoded stream as non-gzip) Decoder support latest version: 4 (decoding as gzip stream) Scenario#3: This is difficult as decoder gets to know the encoded version from the first few bytes of the stream. Which might be zipped or non-zipped, and no reliable way to infer that. Encoded version: 4 (encoded stream is gzip stream) Decoder latest support version: 3 (decoding as non-gzip stream) The change is not backward compatible(Scenario#2 & #3), but depicts the idea for this improvement. It's working well for us. was (Author: asurana): Made the change here: https://github.com/a-surana/kafka/commit/577992015d3bfc5a23e23b5bf32e40a3f92bc74a Scenario#1: This is straightforward, and works with this. Encoded version: 4 Decoder support latest version: 4 Scenario#2: Encoded version: <=3 (encoded stream as non-gzip) Decoder support latest version: 4 (decoding as gzip stream) Scenario#3: This is difficult as decoder gets to know the encoded version from the first few bytes of the stream. Which might be zipped or non-zipped, and no reliable way to infer that. Encoded version: 4 (encoded stream is gzip stream) Decoder latest support version: 3 (decoding as non-gzip stream) The change is not backward compatible(Scenario#2 & #3), but depicts the idea for this improvement. > Reduce assignment data size to improve kafka streams scalability > ---------------------------------------------------------------- > > Key: KAFKA-7149 > URL: https://issues.apache.org/jira/browse/KAFKA-7149 > Project: Kafka > Issue Type: Improvement > Reporter: Ashish Surana > Assignee: Ashish Surana > Priority: Major > > We observed that when we have high number of partitions, instances or > stream-threads, assignment-data size grows too fast and we start getting > below exception at kafka-broker. > RecordTooLargeException > Resolution of this issue is explained at: > https://issues.apache.org/jira/browse/KAFKA-6976 > Still it limits the scalability of kafka streams as moving around 100MBs of > assignment data for each rebalancing affects performance & reliability > (timeout exceptions starts appearing) as well. Also this limits kafka streams > scale even with high max.message.bytes setting as data size increases pretty > quickly with number of partitions, instances or stream-threads. > > Solution: > To address this issue in our cluster, we are sending the compressed > assignment-data. We saw assignment-data size reduced by 8X-10X. This improved > the kafka streams scalability drastically for us and we could now run it with > more than 8,000 partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)