[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168515#comment-17168515 ] ChenLin commented on KAFKA-8328: Thanks for your attention, I will re submit the batch to the trunk version。[~roncenzhao] [~xilangyan] > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168514#comment-17168514 ] ChenLin commented on KAFKA-8328: Thanks for your attention, I will re submit the batch to the trunk version。 > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941695#comment-16941695 ] Jeffrey(Xilang) Yan commented on KAFKA-8328: Hi [~LordChen], do you have code that can work on master branch? > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916292#comment-16916292 ] roncenzhao commented on KAFKA-8328: --- Hi [~LordChen] , what's the progress about this KIP? The KIP page is outdated now. > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865488#comment-16865488 ] ChenLin commented on KAFKA-8328: [~Yohan123] look forward to your reply，KIP url:[https://cwiki.apache.org/confluence/display/SHARDINGSPHERE/KIP-477%3AKafka+smooth+expansion] > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854411#comment-16854411 ] ChenLin commented on KAFKA-8328: Thank you for your attention, I will submit KIP as soon as possible. > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851303#comment-16851303 ] Richard Yu commented on KAFKA-8328: --- Hi [~LordChen] I believe for a change like this, since it involves (I think, public API behavior), it would require a KIP. > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, Kafka_smooth_expansion.patch, > producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !producerP999.png! > !DiskUtil.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8328) Kafka smooth expansion
[ https://issues.apache.org/jira/browse/KAFKA-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834434#comment-16834434 ] ASF GitHub Bot commented on KAFKA-8328: --- lordcheng10 commented on pull request #6690: [KAFKA-8328] Kafka smooth expansion URL: https://github.com/apache/kafka/pull/6690 When expanding the kafka cluster, the new follower will read the data from the earliest offset. This can result in a large amount of historical data being read from the disk, putting a lot of pressure on the disk and affecting the performance of the kafka service, for example, the producer write latency will increase. In general, kafka's original expansion mechanism has the following problems: 1. The new follower will put a lot of pressure on the disk; 2. Causes the producer write latency to increase; 3. Causes the consumer read latency to increase; In order to solve these problems, we have proposed a solution for smooth expansion. The main idea of the scheme is that the newly added follower reads data from the HW position, and when the newly added follower reads the data to a certain time threshold or data size threshold, the follower enters the ISR queue. . Since the new follower reads data from the HW location, most of the data read is in the operating system's cache, so it does not put pressure on the disk and does not affect the performance of the kafka service, thus solving the above problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kafka smooth expansion > -- > > Key: KAFKA-8328 > URL: https://issues.apache.org/jira/browse/KAFKA-8328 > Project: Kafka > Issue Type: Improvement > Components: clients, core >Affects Versions: 0.10.2.0 >Reporter: ChenLin >Priority: Major > Labels: Kafka, expansion > Fix For: 0.10.2.0 > > Attachments: DiskUtil.png, producerP999.png > > > When expanding the kafka cluster, the new follower will read the data from > the earliest offset. This can result in a large amount of historical data > being read from the disk, putting a lot of pressure on the disk and affecting > the performance of the kafka service, for example, the producer write latency > will increase. In general, kafka's original expansion mechanism has the > following problems: > 1. The new follower will put a lot of pressure on the disk; > 2. Causes the producer write latency to increase; > 3. Causes the consumer read latency to increase; > In order to solve these problems, we have proposed a solution for > smooth expansion. The main idea of the scheme is that the newly added > follower reads data from the HW position, and when the newly added follower > reads the data to a certain time threshold or data size threshold, the > follower enters the ISR queue. . Since the new follower reads data from the > HW location, most of the data read is in the operating system's cache, so it > does not put pressure on the disk and does not affect the performance of the > kafka service, thus solving the above problems. > In order to illustrate the problems of the original expansion scheme, > we have done some tests, and there are corresponding test charts in the > attachment. > !image-2019-05-07-14-12-41-314.png! > !image-2019-05-07-14-12-03-005.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
