Sorry I missed that comment on the thread. Proposal looks great, thanks,
Abhijeet!
On Sat, 16 Mar 2024 at 13:19, Abhijeet Kumar
wrote:
> Hi Jorge,
>
> The configs name was chosen to keep it consistent with the other existing
> quota configs, such as
>
Hi Jorge,
The configs name was chosen to keep it consistent with the other existing
quota configs, such as
*replica.alter.log.dirs.io.max.bytes.per.second* as pointed out by Jun in
the thread.
Also, we can revisit the names of the components during implementation,
since those are not exposed to
Hi Abhijeet,
Thanks for the KIP! Looks good to me. I just have a minor comments on
naming:
Would it be work to align the config names to existing quota names?
e.g. `remote.log.manager.copy.byte.rate.quota` (or similar) instead of
`remote.log.manager.copy.max.bytes.per.second`?
Same for new
Thank you all for your comments. As all the comments in the thread are
addressed, I am starting a Vote thread for the KIP. Please have a look.
Regards.
On Thu, Mar 7, 2024 at 12:34 PM Luke Chen wrote:
> Hi Abhijeet,
>
> Thanks for the update and the explanation.
> I had another look, and it
Hi Abhijeet,
Thanks for the update and the explanation.
I had another look, and it LGTM now!
Thanks.
Luke
On Tue, Mar 5, 2024 at 2:50 AM Jun Rao wrote:
> Hi, Abhijeet,
>
> Thanks for the reply. Sounds good to me.
>
> Jun
>
>
> On Sat, Mar 2, 2024 at 7:40 PM Abhijeet Kumar
> wrote:
>
> > Hi
Hi, Abhijeet,
Thanks for the reply. Sounds good to me.
Jun
On Sat, Mar 2, 2024 at 7:40 PM Abhijeet Kumar
wrote:
> Hi Jun,
>
> Thanks for pointing it out. It makes sense to me. We can have the following
> metrics instead. What do you think?
>
>- remote-(fetch|copy)-throttle-time-avg (The
Hi Jun,
Thanks for pointing it out. It makes sense to me. We can have the following
metrics instead. What do you think?
- remote-(fetch|copy)-throttle-time-avg (The average time in ms remote
fetches/copies was throttled by a broker)
- remote-(fetch|copy)-throttle-time--max (The maximum
Hi, Abhijeet,
Thanks for the reply.
The issue with recording the throttle time as a gauge is that it's
transient. If the metric is not read immediately, the recorded value could
be reset to 0. The admin won't realize that throttling has happened.
For client quotas, the throttle time is tracked
Hi Jun,
Clarified the meaning of the two metrics. Also updated the KIP.
kafka.log.remote:type=RemoteLogManager, name=RemoteFetchThrottleTime -> The
duration of time required at a given moment to bring the observed fetch
rate within the allowed limit, by preventing further reads.
Hi, Abhijeet,
Thanks for the explanation. Makes sense to me now.
Just a minor comment. Could you document the exact meaning of the following
two metrics? For example, is the time accumulated? If so, is it from the
start of the broker or over some window?
kafka.log.remote:type=RemoteLogManager,
Hi Jun,
The existing quota system for consumers is designed to throttle the
consumer if it exceeds the allowed fetch rate.
The additional quota we want to add works on the broker level. If the
broker-level remote read quota is being
exceeded, we prevent additional reads from the remote storage
Hi, Abhijeet,
Sorry for the late reply. It seems that you haven't updated the KIP based
on the discussion? One more comment.
11. Currently, we already have a quota system for both the producers and
consumers. I can understand why we need an additional
remote.log.manager.write.quota.default
Comments inline
On Wed, Dec 6, 2023 at 1:12 AM Jun Rao wrote:
> Hi, Abhijeet,
>
> Thanks for the KIP. A few comments.
>
> 10. remote.log.manager.write.quota.default:
> 10.1 For other configs, we
> use replica.alter.log.dirs.io.max.bytes.per.second. To be consistent,
> perhaps this can be sth
Comments inline
On Tue, Nov 28, 2023 at 3:50 PM Luke Chen wrote:
>
> Hi Abhijeet,
>
> Thanks for the KIP!
> This is an important feature for tiered storage.
>
> Some comments:
> 1. Will we introduce new metrics for this tiered storage quotas?
> This is important because the admin can know the
Yes, it is part of V1. I have added it now.
I will follow up on the KIP comments in the next couple of days and
try to close the review in the next few weeks.
Regards.
On Thu, Feb 1, 2024 at 7:40 PM Francois Visconte
wrote:
>
> Hi,
>
> I see that the ticket has been left untouched since a
Hi,
I see that the ticket has been left untouched since a while now.
Should it be included in the tiered storage v1?
We've observed that lacking a way to throttle uploads to tiered storage has
a major impact on
producers and consumers when tiered storage access recovers (starving disk
Hi, Abhijeet,
Thanks for the KIP. A few comments.
10. remote.log.manager.write.quota.default:
10.1 For other configs, we
use replica.alter.log.dirs.io.max.bytes.per.second. To be consistent,
perhaps this can be sth like remote.log.manager.write.max.bytes.per.second.
10.2 Could we list the new
Hi Abhijeet,
Thanks for the KIP!
This is an important feature for tiered storage.
Some comments:
1. Will we introduce new metrics for this tiered storage quotas?
This is important because the admin can know the throttling status by
checking the metrics while the remote write/read are slow, like
Hi All,
I have created KIP-956 for defining read and write quota for tiered storage.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas
Feedback and suggestions are welcome.
Regards,
Abhijeet.
19 matches
Mail list logo
