Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-15 Thread 太上玄元道君
> Perhaps clarify, they can call: > getStats(subscriptionBacklogSize=true, getPreciseBacklog=false, > getEarliestTimeInBacklog=false) > Each subscription will contain backlogSize. > The subscription with max backlogSize will also be the one with oldest age. Yes, "The subscription with max backlog

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-15 Thread Asaf Mesika
> topic > > > > > > > > backlog. > > > > > > > > > > There are two types of quotas: > > > > > > > > > > * Size based: The limit is for the topic backlog size (as > > we > &g

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-14 Thread 太上玄元道君
nacknowledged messages will be > > > > > > acknowledged > > > > > > > > in > > > > > > > > > order until both the topic backlog size or age will fall > > inside > > > > the > > &g

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-14 Thread Asaf Mesika
cy. > > > > > > > > > > > > > > > > The user today can calculate quota used for size based limit, > > > since > > > > > > there > > > > > > > > are two metrics that are exposed

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-09 Thread 太上玄元道君
ar_backlog_size_quota_used_percentage` > > > > > > > `pulsar_backlog_time_quota_used_percentage` > > > > > > > > > > > > > > You would like to know what triggered the alert, hence two. > > > > > &

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-09 Thread Asaf Mesika
ot correct. > > > > > > It checks against the topic / namespace policy, and if it doesn't > > > > exist, > > > > > it > > > > > > falls back on the default configuration key mentioned above. > > > > > > > >

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-08 Thread 太上玄元道君
gt; > >. > > > > > > So in general, we don't need to worry about it will lead to I/O. > > > > > > > > > > > > > > > I'm afraid of that. > > > > > Today the quota is checked periodically, right? So that's how the > &

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-06 Thread Asaf Mesika
ries. Since > > > > Prometheus flush only every 2 hours, or any there TSDB, it will cost > > you > > > > too much. > > > > > > > > I suggest exposing the name via the topic stats. This way they can > > issue > > > a > > > &

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-06 Thread 太上玄元道君
; > will > > > > be > > > > > > triggered. > > > > > > > > > > This seems like default values, not the actual values. Can you > please > > > > > provide an explanation in the PIP and link to read more: > > > > > 1

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-05 Thread Asaf Mesika
f I/O) and there is an > estimate > > of > > > > it. I presume you would want to use the estimated one, is that > correct? > > > > The backlog quota itself, uses the accurate or the estimated when it > > > starts > > > > evicting entries (

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-02 Thread 太上玄元道君
al section, you specify that your goal is to add a > > "proximity" > > > metric. > > > a) You must define that - what is proximity metric exactly? What are > its > > > units? How are you planning to calculate it? > > > b) Proximity is not a good term

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-01 Thread PengHui Li
from now). How do you plan to calculate the current age of the > > > earliest message without bearing that I/O cost on each metric > > calculation? > > > > > > 3. In the Goal section, you specify that your goal is to add a > > "proximity" > >

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-03-01 Thread Asaf Mesika
n help provide alternative names. > > > > 4. Maybe we should provide the used quota percentage for both limits, > > instead of one per both, since it's easier to act upon the alert when you > > need which one triggered it. > > > > 5. I didn't underst

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-02-27 Thread 太上玄元道君
quota percentage for both limits, > > instead of one per both, since it's easier to act upon the alert when you > > need which one triggered it. > > > > 5. I didn't understand the "slowest_subscription" label used when

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-02-27 Thread 太上玄元道君
I suggest writing a "High Level Design" section, and add everything you > need to know for this proposal, so I don't need to read the > implementation details below (code). > > Thanks, > > Asaf > > > On Wed, Feb 22, 2023 at 4:52 PM 太上玄

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-02-27 Thread PengHui Li
quot; section, and add everything you > need to know for this proposal, so I don't need to read the > implementation details below (code). > > Thanks, > > Asaf > > > On Wed, Feb 22, 2023 at 4:52 PM 太上玄元道君 wrote: > > > Hi all, > > > > I've start

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-02-26 Thread Asaf Mesika
iting a "High Level Design" section, and add everything you need to know for this proposal, so I don't need to read the implementation details below (code). Thanks, Asaf On Wed, Feb 22, 2023 at 4:52 PM 太上玄元道君 wrote: > Hi all, > > I've started a PIP to discuss:

[Discuss] PIP-248: Add backlog eviction metric

2023-02-22 Thread 太上玄元道君
Hi all, I've started a PIP to discuss: PIP-248 Add backlog eviction metric ### Motivation: Pulsar has 2 configurations for the backlog eviction: `backlogQuotaDefaultLimitBytes` and `backlogQuotaDefaultLimitSecond`, if topic backlog reaches the threshold of any item, backlog eviction wi