[
https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChenLin updated KAFKA-8328:
---------------------------
Attachment: Kafka_smooth_expansion.patch
> Kafka smooth expansion
> ----------------------
>
> Key: KAFKA-8328
> URL: https://issues.apache.org/jira/browse/KAFKA-8328
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core
> Affects Versions: 0.10.2.0
> Reporter: ChenLin
> Priority: Major
> Labels: Kafka, expansion
> Fix For: 0.10.2.0
>
> Attachments: DiskUtil.png, Kafka_smooth_expansion.patch,
> producerP999.png
>
>
> When expanding the kafka cluster, the new follower will read the data from
> the earliest offset. This can result in a large amount of historical data
> being read from the disk, putting a lot of pressure on the disk and affecting
> the performance of the kafka service, for example, the producer write latency
> will increase. In general, kafka's original expansion mechanism has the
> following problems:
> 1. The new follower will put a lot of pressure on the disk;
> 2. Causes the producer write latency to increase;
> 3. Causes the consumer read latency to increase;
> In order to solve these problems, we have proposed a solution for
> smooth expansion. The main idea of the scheme is that the newly added
> follower reads data from the HW position, and when the newly added follower
> reads the data to a certain time threshold or data size threshold, the
> follower enters the ISR queue. . Since the new follower reads data from the
> HW location, most of the data read is in the operating system's cache, so it
> does not put pressure on the disk and does not affect the performance of the
> kafka service, thus solving the above problems.
> In order to illustrate the problems of the original expansion scheme,
> we have done some tests, and there are corresponding test charts in the
> attachment.
> !image-2019-05-07-14-12-41-314.png!
> !image-2019-05-07-14-12-03-005.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
