[jira] [Assigned] (KAFKA-16043) Add Quota configuration for topics
[ https://issues.apache.org/jira/browse/KAFKA-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Afshin Moazami reassigned KAFKA-16043: -- Assignee: Afshin Moazami > Add Quota configuration for topics > -- > > Key: KAFKA-16043 > URL: https://issues.apache.org/jira/browse/KAFKA-16043 > Project: Kafka > Issue Type: New Feature >Reporter: Afshin Moazami >Assignee: Afshin Moazami >Priority: Major > > To be able to have topic-partition quota, we need to introduce two topic > configuration for the producer-byte-rate and consumer-byte-rate. > The assumption is that all partitions of the same topic get the same quota, > so we define one config per topic. > This configuration should work both with zookeeper and kraft setup. > Also, we should define a default quota value (to be discussed) and > potentially use the same format as user/client default configuration using > `` as the value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-16044) Throttling using Topic Partition Quota
[ https://issues.apache.org/jira/browse/KAFKA-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Afshin Moazami reassigned KAFKA-16044: -- Assignee: Afshin Moazami > Throttling using Topic Partition Quota > --- > > Key: KAFKA-16044 > URL: https://issues.apache.org/jira/browse/KAFKA-16044 > Project: Kafka > Issue Type: New Feature >Reporter: Afshin Moazami >Assignee: Afshin Moazami >Priority: Major > > With > !https://issues.apache.org/jira/secure/viewavatar?size=xsmall=21141=issuetype! > KAFKA-16042 introducing the topic-partition byte rate and metrics, and > !https://issues.apache.org/jira/secure/viewavatar?size=xsmall=21141=issuetype! > KAFKA-16043 introducing the quota limit configuration in the topic-level, we > can enforce quota on topic-partition level for configured topics. > More details in the > [KIP-1010|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799504#comment-17799504
]
Afshin Moazami commented on KAFKA-10897:
For topic-partition quota configuration, I proposed this KIP:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota]
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Assignee: Kahn Cheny
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16044) Throttling using Topic Partition Quota
Afshin Moazami created KAFKA-16044: -- Summary: Throttling using Topic Partition Quota Key: KAFKA-16044 URL: https://issues.apache.org/jira/browse/KAFKA-16044 Project: Kafka Issue Type: New Feature Reporter: Afshin Moazami With !https://issues.apache.org/jira/secure/viewavatar?size=xsmall=21141=issuetype! KAFKA-16042 introducing the topic-partition byte rate and metrics, and !https://issues.apache.org/jira/secure/viewavatar?size=xsmall=21141=issuetype! KAFKA-16043 introducing the quota limit configuration in the topic-level, we can enforce quota on topic-partition level for configured topics. More details in the [KIP-1010|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16043) Add Quota configuration for topics
Afshin Moazami created KAFKA-16043: -- Summary: Add Quota configuration for topics Key: KAFKA-16043 URL: https://issues.apache.org/jira/browse/KAFKA-16043 Project: Kafka Issue Type: New Feature Reporter: Afshin Moazami To be able to have topic-partition quota, we need to introduce two topic configuration for the producer-byte-rate and consumer-byte-rate. The assumption is that all partitions of the same topic get the same quota, so we define one config per topic. This configuration should work both with zookeeper and kraft setup. Also, we should define a default quota value (to be discussed) and potentially use the same format as user/client default configuration using `` as the value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16042) Quota Metrics based on per topic-partition produce/fetch byte rate
Afshin Moazami created KAFKA-16042: -- Summary: Quota Metrics based on per topic-partition produce/fetch byte rate Key: KAFKA-16042 URL: https://issues.apache.org/jira/browse/KAFKA-16042 Project: Kafka Issue Type: New Feature Components: core Reporter: Afshin Moazami Assignee: Afshin Moazami Currently, Kafka emits the producer-byte-rate and fetch-bytes-rate for quota calculations. By adding a new signature to the `[quotaMetricTags|https://github.com/afshing/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/server/quota/ClientQuotaCallback.java#L40]` method to add the individual topic-partitions size as a parameter, we can define metrics based on the topic name and partition id. To do that, we need both `ProduceRequest` and `FetchResponse` have the `partitionSizes` method and it is public. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-10897) kafka quota optimization
[
https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727591#comment-17727591
]
Afshin Moazami commented on KAFKA-10897:
Hey team,
I am new to the community.
We are implementing topic-partition quota at Salesforce fork of Kafka, where
you can define the produce/consume byte rate per topic-partition. I believe
that feature is addressing the third suggestion.
Is there a process to suggest that change, and maybe review the design?
> kafka quota optimization
>
>
> Key: KAFKA-10897
> URL: https://issues.apache.org/jira/browse/KAFKA-10897
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, config, consumer, core
>Affects Versions: 2.7.0
>Reporter: yangyijun
>Assignee: Kahn Cheny
>Priority: Blocker
>
> *1.The current quota dimensions is as follows:*
> {code:java}
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/users//clients/
> /config/users//clients/
> /config/users/
> /config/clients/
> /config/clients/{code}
> *2. Existing problems:*
>
> {code:java}
> 2.1.The quota dimensions is not fine enough.
> 2.2.When multiple users on the same broker produce and consume a large amount
> of data at the same time, if you want the broker to run normally, you must
> make the sum of all user quota byte not exceed the upper throughput limit of
> the broker.
> 2.3.Even if all the user rate does not reach the upper limit of the broker,
> but all the user rate is concentrated on a few disks and exceeds the
> read-write load of the disk, all the produce and consume requests will be
> blocked.
> 2.4.Sometimes it's just one topic rate increase sharply under the user, so we
> just need to limit the increase sharply topics.
> {code}
>
> *3. Suggestions for improvement*
> {code:java}
> 3.1. Add the upper limit of single broker quota byte.
> 3.2. Add the upper limit of single disk quota byte on the broker.
> 3.3. Add topic quota dimensions.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
