Closing this discussion thread and the PIP. Apart from the discussion
present in this thread, I presented the detailed requirements in a dev meet
on 23rd November and the conclusion was that we will actually go ahead and
implement the requirements in pulsar itself.
There was a pre-requisite of
I've captured our requirements in detail in this document -
https://docs.google.com/document/d/1-y5nBaC9QuAUHKUGMVVe4By-SmMZIL4w09U1byJBbMc/edit
Added it to agenda document as well. Will join the meeting and discuss.
Regards
On Wed, Nov 22, 2023 at 10:49 PM Lari Hotari wrote:
> I have written
I have written a long blog post that contains the context, the summary
of my view point about PIP-310 and the proposal for proceeding:
https://codingthestreams.com/pulsar/2023/11/22/pulsar-slos-and-rate-limiting.html
Let's discuss this tomorrow in the Pulsar community meeting [1]. Let's
Hi Girish,
replies inline and after that there are some updates about my
preparation for the community meeting on Thursday. (there's
https://github.com/lhotari/async-tokenbucket with a PoC for a
low-level high performance token bucket implementation)
On Sat, 11 Nov 2023 at 17:25, Girish Sharma
One final reply from me before the holidays :)
On Sat, Nov 11, 2023 at 4:00 PM Lari Hotari wrote:
> Hi Girish,
>
> replies inline.
>
> > Hello Lari, replies inline. It's festive season here so I might be late
> in
> > the next reply.
>
> I'll have limited availability next week so possibly not
Hi Girish,
replies inline.
> Hello Lari, replies inline. It's festive season here so I might be late in
> the next reply.
I'll have limited availability next week so possibly not replying
until the following week. We have the next Pulsar community meeting on
November 23rd, so let's wrap up all
Hello Lari, replies inline. It's festive season here so I might be late in
the next reply.
On Fri, Nov 10, 2023 at 4:51 PM Lari Hotari wrote:
> Hi Girish,
>
> >
> >
> > > tokens would be added to the bucket as are consumed by the actual
> > > traffic. The left over tokens 10MB - actual rate
Hi Girish,
replies inline.
> > I'd pick the first bucket for handling the 10MB rate.
> > The capacity of the first bucket would be 15MB * 120=1800MB. The fill
> > would happen in special way. I'm not sure if Bucket4J has this at all.
> > So describing the way of adding tokens to the bucket: the
Hello Lari, replies inline
On Thu, Nov 9, 2023 at 6:50 AM Lari Hotari wrote:
> Hi Girish,
>
> replies inline.
>
> On Thu, 9 Nov 2023 at 00:29, Girish Sharma
> wrote:
> > While dual-rate dual token bucket looks promising, there is still some
> > challenge with respect to allowing a certain peak
Hi Girish,
replies inline.
On Thu, 9 Nov 2023 at 00:29, Girish Sharma wrote:
> While dual-rate dual token bucket looks promising, there is still some
> challenge with respect to allowing a certain peak burst for/up to a bigger
> duration. I am explaining it below:
> Assume a 10MBps topic.
Hi Girish,
Replies inline.
> > The current state of rate limiting is not acceptable in Pulsar. We
> > need to fix things in the core.
> >
>
> I wouldn't say that it's not acceptable. The precise one works as expected
> as a basic rate limiter. Its only when there are complex requirements, the
>
Hello Lari,
I've now gone through a bunch of rate limiting algorithms, along with dual
rate, dual bucket algorithm.
Reply inline.
On Tue, Nov 7, 2023 at 11:32 PM Lari Hotari wrote:
>
> Bucket4J documentation gives some good ideas and it's shows how the
> token bucket algorithm could be varied.
Hello Lari, while I am yet to reply to your yesterday''s email, I am trying
to wrap this discussion about the need of pluggable rate limiter, so
replying to this first.
Comments inline
On Wed, Nov 8, 2023 at 5:35 PM Lari Hotari wrote:
> Hi Girish,
>
> > On Asaf's comment on too many public
Hi Girish,
> On Asaf's comment on too many public interfaces in Pulsar and no other
> Apache software having so many public interfaces - I would like to ask, has
> that brought in any con though? For this particular use case, I feel like
> having it has a public interface would actually improve
Hi Rajan,
Thank you for sharing your opinion. It appears you are in favor of
pluggable interfaces for rate limiters. I would like to offer a
perspective on why we should defer the pluggability aspect of rate
limiters. If there is a real need, it can be considered later.
Currently, there's a
Hello Rajan,
I haven't updated the PIP with a better interface for PublishRateLimiter
yet as the discussion here in this thread went in a different direction.
Personally, I agree with you that even if we choose one algorithm and
improve the built-in rate limiter, it still may not suit all use
Hi Lari/Girish,
I am sorry for jumping late in the discussion but I would like to
acknowledge the requirement of pluggable publish rate-limiter and I had
also asked it during implementation of publish rate limiter as well. There
are trade-offs between different rate-limiter implementations based
Hi Girish,
Replies inline.
On Tue, 7 Nov 2023 at 15:26, Girish Sharma wrote:
>
> Hello Lari, replies inline.
>
> I will also be going through some textbook rate limiters (the one you shared,
> plus others) and propose the one that at least suits our needs in the next
> reply.
sounds good.
I just want to add one thing to the mix here.
You can see by the amount of plugin interfaces Pulsar has, somebody "left
the door open" for too long.
You can agree with me that the number of those interfaces is not normal for
any open source software. I know HBase for example, or Kafka - never
Hello Lari, replies inline.
I will also be going through some textbook rate limiters (the one you
shared, plus others) and propose the one that at least suits our needs in
the next reply.
On Tue, Nov 7, 2023 at 2:49 PM Lari Hotari wrote:
>
> It is bi-weekly on Thursdays. The meeting calendar,
Hi Girish,
I think we are starting to get into the concrete details of rate
limiters and how we could start improving the existing feature.
It is very helpful that you are sharing your insight and experience of
operating Pulsar at scale.
Replies inline.
On Mon, 6 Nov 2023 at 15:37, Girish Sharma
Hello Lari, inline once again.
On Mon, Nov 6, 2023 at 5:44 PM Lari Hotari wrote:
> Hi Girish,
>
> Replies inline. We are getting into a very detailed discussion. We
> could also discuss this topic in one of the upcoming Pulsar Community
> meetings. However, I might miss the next meeting that
Hi Girish,
Replies inline. We are getting into a very detailed discussion. We
could also discuss this topic in one of the upcoming Pulsar Community
meetings. However, I might miss the next meeting that is scheduled
this Thursday.
Although I am currently opposing to your proposal PIP-310, I am
On Sat, Nov 4, 2023 at 9:21 PM Lari Hotari wrote:
> One additional question:
>
> In your use case, do you have multiple producers concurrently producing to
> the same topic from different clients?
>
> The use case is challenging in the current implementation when using topic
> producing rate
Replies inline.
On Sat, Nov 4, 2023 at 8:55 PM Lari Hotari wrote:
>
> One possibility would be to improve the existing rate limiter to allow
> bursting.
> I think that Pulsar's out-of-the-box rate limiter should cover 99% of the
> use cases instead of having one implementing their own rate
One additional question:
In your use case, do you have multiple producers concurrently producing to
the same topic from different clients?
The use case is challenging in the current implementation when using topic
producing rate limiting. The problem is that the different producers will
be able
Replies inline
On Fri, 3 Nov 2023 at 20:48, Girish Sharma wrote:
> Could you please elaborate more on these details? Here are some questions:
> > 1. What do you mean that it is too strict?
> > - Should the rate limiting allow bursting over the limit for some
> time?
> >
>
> That's one of
Hello Lari, replies inline.
On Fri, Nov 3, 2023 at 11:13 PM Lari Hotari wrote:
> Hi Girish,
>
> Thanks for the questions. I'll reply to them
>
> > does this sharing of the same tcp/ip connection happen across partitions
> as
> > well (assuming both the partitions of the topic are on the same
Hi Girish,
Thanks for the questions. I'll reply to them
> does this sharing of the same tcp/ip connection happen across partitions as
> well (assuming both the partitions of the topic are on the same broker)?
> i.e. producer 127.0.0.1 for partition
> `persistent://tenant/ns/topic0-partition0`
Hi Tao,
You seemed to miss sending the link that you were referring to. Are you
referring to
the thread "[discuss] Support fail-fast strategy when broker rate-limited" [1] ?
-Lari
1 - https://lists.apache.org/thread/tp2f1ghomj2kw5ltgz8w8k5s58gs88qz
On 2023/11/03 12:11:31 太上玄元道君 wrote:
> Hi
Hello Tao,
As I understand, there is a fine balance between rate-limiting,
backpressure and not keeping clients waiting. Different use cases may need
different approach to rate-limiting and thus, making rate limiter
customizable is my first step towards making pulsar more customizable as
per need.
Hi Girish,
There is also a discussion thread[1] about rate-limiting.
I think there is some conflicts between some kind of rate-limiter and
backpressure
Take the fail-fast strategy as an example:
Brokers have to reply to clients after receiving and decode the message,
but the broker also has the
Hello Lari,
Thanks for bringing this to my attention. I went through the links, but
does this sharing of the same tcp/ip connection happen across partitions as
well (assuming both the partitions of the topic are on the same broker)?
i.e. producer 127.0.0.1 for partition
Hi Girish,
In order to address your problem described in the PIP document [1], it might be
necessary to make improvements in how rate limiters apply backpressure in
Pulsar.
Pulsar uses mainly TCP/IP connection level controls for achieving backpressure.
The challenge is that Pulsar can share a
Replied in PR.
On Thu, Oct 19, 2023 at 3:51 PM Girish Sharma
wrote:
> Hi,
> Currently, there are only 2 kinds of publish rate limiters - polling based
> and precise. Users have an option to use either one of them in the topic
> publish rate limiter, but the resource group rate limiter only
Hi,
Currently, there are only 2 kinds of publish rate limiters - polling based
and precise. Users have an option to use either one of them in the topic
publish rate limiter, but the resource group rate limiter only uses polling
one.
There are challenges with both the rate limiters and the fact
36 matches
Mail list logo
