[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799504#comment-17799504
]
Afshin Moazami commented on KAFKA-10897:
For topic-partition quota configuration, I proposed this KIP:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota]
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Assignee: Kahn Cheny
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728074#comment-17728074
]
Omnia Ibrahim commented on KAFKA-10897:
---
Hi [~afshing] to propose any API changes you need to write a Kafka Improvement
Proposal (known as KIP) you can find the details here
[https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals]
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Assignee: Kahn Cheny
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727591#comment-17727591
]
Afshin Moazami commented on KAFKA-10897:
Hey team,
I am new to the community.
We are implementing topic-partition quota at Salesforce fork of Kafka, where
you can define the produce/consume byte rate per topic-partition. I believe
that feature is addressing the third suggestion.
Is there a process to suggest that change, and maybe review the design?
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Assignee: Kahn Cheny
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389918#comment-17389918
]
yangyijun commented on KAFKA-10897:
---
Dear [~KahnCheny] , will you start this part of the work?
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389913#comment-17389913
]
yangyijun commented on KAFKA-10897:
---
It has not started yet
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389911#comment-17389911
]
yangyijun commented on KAFKA-10897:
---
Not yet
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389732#comment-17389732
]
Kahn Cheny commented on KAFKA-10897:
Is this feature scheduled?
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
