Andrey Neporada created KAFKA-3979: -------------------------------------- Summary: Optimize memory used by replication process by using adaptive fetch message size Key: KAFKA-3979 URL: https://issues.apache.org/jira/browse/KAFKA-3979 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.10.0.0 Reporter: Andrey Neporada
Current replication process fetches messages in replica.fetch.max.bytes-sized chunks. Since replica.fetch.max.bytes should be bigger than max.message.bytes for replication to work, one can face big memory consumption for replication process, especially for installations with big number of partitions. Proposed solution is to try to fetch messages in smaller chunks (say replica.fetch.base.bytes). If we encounter message bigger than current fetch chunk, we increase chunk (f.e. twofold) and retry. After replicating this bigger message, we shrunk fetch chunk size back until it reaches replica.fetch.base.bytes replica.fetch.base.bytes should be chosen big enough not to affect throughput and to be bigger than most of messages. However, it can be much less than replica.fetch.max.bytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)