Re: [VOTE] PIP-265: PR-based system for managing and reviewing PIPs
+1 (non-binding) Thanks for driving this helpful proposal! Best, Max Xu On Wed, May 10, 2023 at 6:52 PM Asaf Mesika wrote: > Hi, > > I'm starting the vote process for PIP-265. > > Link: https://github.com/apache/pulsar/issues/20207 > > Thanks! > > Asaf >
Re: [VOTE] PIP-265: PR-based system for managing and reviewing PIPs
+1 (binding) It will be helpful for PR review. Best, Hang PengHui Li 于2023年5月15日周一 10:09写道： > > +1 (binding) > > Thanks, > Penghui > > On Wed, May 10, 2023 at 6:52 PM Asaf Mesika wrote: > > > Hi, > > > > I'm starting the vote process for PIP-265. > > > > Link: https://github.com/apache/pulsar/issues/20207 > > > > Thanks! > > > > Asaf > >
Re: [VOTE] PIP-265: PR-based system for managing and reviewing PIPs
+1 (binding) Thanks, Penghui On Wed, May 10, 2023 at 6:52 PM Asaf Mesika wrote: > Hi, > > I'm starting the vote process for PIP-265. > > Link: https://github.com/apache/pulsar/issues/20207 > > Thanks! > > Asaf >
Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api
There are multiple factors tcp would be better than http when you are considering performance and scale due to lower overhead, faster parsing, more control over connection, congestion control, etc and that could be reasons why we have tcp binary protocol for producers/consumers communication with server and this will be an extension for those producers and consumers applications which would like to periodically or on-demand retrieve stats from large number of application nodes and we would like to let those producers/consumers retrieve it and use it in application without worrying about scaling issues at admin apis. Thanks, Rajan On Sun, May 14, 2023 at 6:47 AM Asaf Mesika wrote: > If I dive into the exact details of the cause of the > performance implications in current Admin HTTP API: > > Do you think the root cause of the performance is the Jersey implementation > of `AsyncResponse.resume(stats)` which takes a thread from a thread pool, > serialize the object and then performs a blocking I/O write of the JSON > string? > > If I compared that with Netty HTTP, it would write the same String using > async I/O and not blocking a thread. > > Given the normally large response size of those objects, the response > headers or request headers are negligible in terms of performance impact. > > In terms of accepting connection, both Netty and Jetty has async IO > implementation. > > Compared Jetty and Jersey with Netty based binary TCP, if we end up writing > a JSON string, the only difference I see is the blocking I/O of writing the > response. > > WDYT? > > > On Fri, May 12, 2023 at 7:29 AM Rajan Dhabalia > wrote: > > > Communicating over binary protocol is more scalable and performant than > > HTTP. Admin API over http has a long history of bottleneck and > performance > > issues which could also sometimes be a bottleneck for lookup requests and > > that was the reason we introduced lookup over binary protocol as well. We > > have multiple usecases which require fetching stats with relatively > higher > > rate and definitely we would like to avoid it over http which could be a > > bottleneck for those applications or could be for others. > > This PIP doesn't mention security so, let's not misinterpret the > usecases. > > Sometimes, pulsar is deployed behind the proxy and potentially used SNI > > routing proxy which can't be used as http proxy and we would like to let > > users access stats for a given topic using the same broker-service url > > rather than having separate http endpoints. So, this api addresses scale, > > performance, and use accessibility in pulsar. > > > > Thanks, > > Rajan > > > > On Thu, May 11, 2023 at 6:24 AM Asaf Mesika > wrote: > > > > > Before I dive into the PIP, I have several questions on the background > > > provided below: > > > > > > > > > On Tue, May 9, 2023 at 9:08 AM Rajan Dhabalia > > > wrote: > > > > > > > Hi, > > > > > > > > Right now, Pulsar provides the topic's stats and stats-internal over > > HTTP > > > > admin API, and this stats data is used by user applications and also > by > > > > Pulsar internal components such as Pulsar-functions to derive the > > certain > > > > states of the applications. > > > > for example, there are use cases where the application wants to check > > the > > > > topic's backlog, subscription's state (readPosition, list of > > > > subscriptions), numberOfEntriesSinceFirstNotAckedMessage, etc to > > > bootstrap > > > > the application or handle the application’s resiliency and state > > > > dynamically. Applications can retrieve this stats information by > using > > > the > > > > broker’s admin HTTP APIs. > > > > > > > > However, stats retrieval over HTTP API doesn’t work well in use cases > > > when > > > > users would like to access this API at a higher scale when a large > > number > > > > of application nodes would like to use it over HTTP which could > > overload > > > > brokers and sometimes makes broker irresponsive and impact admin API > > > > performance. It also becomes difficult when Pulsar is deployed in the > > > cloud > > > > behind the SNI proxy and applications also want to access large-scale > > > stats > > > > information periodically over different HTTP ports. Instead it would > be > > > > better if applications can fetch stats over on the same binary > protocol > > > for > > > > scalability and accessibility reasons. > > > > > > > > > > Why do you think using a binary protocol over HTTP would make more > > > performant to respond to multiple calls at once? > > > Same question but for the security issue - why do you think the HTTP > port > > > of admin API is harder to access than the binary protocol port? > > > > > > > > > > > > > > > > > > > > Therefore, there are multiple use cases where producer/consumer > > > > applications need stats information for topics using the client > library > > > > over binary protocol. Hence, this PIP introduces client API for > > producers > > > > and consumers to access topic
Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system
> On May 10, 2023, at 1:01 AM, Asaf Mesika wrote: > > On Tue, May 9, 2023 at 11:29 PM Dave Fisher wrote: > >> >> On May 8, 2023, at 2:49 AM, Asaf Mesika wrote: >>> >>> Your feedback made me realized I need to add "TL;DR" section, which I >> just >>> added. >>> >>> I'm quoting it here. It gives a brief summary of the proposal, which >>> requires up to 5 min of read time, helping you get a high level picture >>> before you dive into the background/motivation/solution. >>> >>> -- >>> TL;DR >>> >>> Working with Metrics today as a user or a developer is hard and has many >>> severe issues. >>> >>> From the user perspective: >>> >>> - One of Pulsar strongest feature is "cheap" topics so you can easily >>> have 10k - 100k topics per broker. Once you do that, you quickly learn >> that >>> the amount of metrics you export via "/metrics" (Prometheus style >> endpoint) >>> becomes really big. The cost to store them becomes too high, queries >>> time-out or even "/metrics" endpoint it self times out. >>> The only option Pulsar gives you today is all-or-nothing filtering and >>> very crude aggregation. You switch metrics from topic aggregation >> level to >>> namespace aggregation level. Also you can turn off producer and >> consumer >>> level metrics. You end up doing it all leaving you "blind", looking at >> the >>> metrics from a namespace level which is too high level. You end up >>> conjuring all kinds of scripts on top of topic stats endpoint to glue >> some >>> aggregated metrics view for the topics you need. >>> - Summaries (metric type giving you quantiles like p95) which are used >>> in Pulsar, can't be aggregated across topics / brokers due its inherent >>> design. >>> - Plugin authors spend too much time on defining and exposing metrics >> to >>> Pulsar since the only interface Pulsar offers is writing your metrics >> by >>> your self as UTF-8 bytes in Prometheus Text Format to byte stream >> interface >>> given to you. >>> - Pulsar histograms are exported in a way that is not conformant with >>> Prometheus, which means you can't get the p95 quantile on such >> histograms, >>> making them very hard to use in day to day life. >> >> What version of DataSketches is used to produce the histogram? Is is still >> an old Yahoo one, or are we using an updated one from Apache DataSketches? >> >> Seems like this is a single PR/small PIP for 3.1? > > > Histograms are a list of buckets, each is a counter. > Summary is a collection of values collected over a time window, which at > the end you get a calculation of the quantiles of those values: p95, p50, > and those are exported from Pulsar. > > Pulsar histogram do not use Data Sketches. Bookkeeper Metrics wraps Yahoo DataSketches last I checked. > They are just counters. > They are not adhere to Prometheus since: > a. The counter is expected to be cumulative, but Pulsar resets each bucket > counter to 0 every 1 min > b. The bucket upper range is expected to be written as an attribute "le" > but today it is encoded in the name of the metric itself. > > This is a breaking change, hence hard to mark in any small release. > This is why it's part of this PIP since so many things will break, and all > of them will break on a separate layer (OTel metrics), hence not break > anyone without their consent. If this change will break existing Grafana dashboards and other operational monitoring already in place then it will break guarantees we have made about safely being able to downgrade from a bad upgrade. > > > >> >> >>> - Too many metrics are rates which also delta reset every interval you >>> configure in Pulsar and restart, instead of relying on cumulative (ever >>> growing) counters and let Prometheus use its rate function. >>> - and many more issues >>> >>> From the developer perspective: >>> >>> - There are 4 different ways to define and record metrics in Pulsar: >>> Pulsar own metrics library, Prometheus Java Client, Bookkeeper metrics >>> library and plain native Java SDK objects (AtomicLong, ...). It's very >>> confusing for the developer and create inconsistencies for the end user >>> (e.g. Summary for example is different in each). >>> - Patching your metrics into "/metrics" Prometheus endpoint is >>> confusing, cumbersome and error prone. >>> - many more >>> >>> This proposal offers several key changes to solve that: >>> >>> - Cardinality (supporting 10k-100k topics per broker) is solved by >>> introducing a new aggregation level for metrics called Topic Metric >> Group. >>> Using configuration, you specify for each topic its group (using >>> wildcard/regex). This allows you to "zoom" out to a more detailed >>> granularity level like groups instead of namespaces, which you control >> how >>> many groups you'll have hence solving the cardinality issue, without >>> sacrificing level of detail too much. >>> - Fine-grained filtering mechanism, dynamic. You'll have
Re: [DISCUSS] PIP-264: Enhanced OTel-based metric system
Bumping, as I have received only 1 partial feedback so far on the PIP. Any feedback is highly appreciated! Thanks! Asaf On Wed, May 10, 2023 at 11:00 AM Asaf Mesika wrote: > > > On Tue, May 9, 2023 at 11:29 PM Dave Fisher wrote: > >> >> >> > On May 8, 2023, at 2:49 AM, Asaf Mesika wrote: >> > >> > Your feedback made me realized I need to add "TL;DR" section, which I >> just >> > added. >> > >> > I'm quoting it here. It gives a brief summary of the proposal, which >> > requires up to 5 min of read time, helping you get a high level picture >> > before you dive into the background/motivation/solution. >> > >> > -- >> > TL;DR >> > >> > Working with Metrics today as a user or a developer is hard and has many >> > severe issues. >> > >> > From the user perspective: >> > >> > - One of Pulsar strongest feature is "cheap" topics so you can easily >> > have 10k - 100k topics per broker. Once you do that, you quickly >> learn that >> > the amount of metrics you export via "/metrics" (Prometheus style >> endpoint) >> > becomes really big. The cost to store them becomes too high, queries >> > time-out or even "/metrics" endpoint it self times out. >> > The only option Pulsar gives you today is all-or-nothing filtering and >> > very crude aggregation. You switch metrics from topic aggregation >> level to >> > namespace aggregation level. Also you can turn off producer and >> consumer >> > level metrics. You end up doing it all leaving you "blind", looking >> at the >> > metrics from a namespace level which is too high level. You end up >> > conjuring all kinds of scripts on top of topic stats endpoint to glue >> some >> > aggregated metrics view for the topics you need. >> > - Summaries (metric type giving you quantiles like p95) which are used >> > in Pulsar, can't be aggregated across topics / brokers due its >> inherent >> > design. >> > - Plugin authors spend too much time on defining and exposing metrics >> to >> > Pulsar since the only interface Pulsar offers is writing your metrics >> by >> > your self as UTF-8 bytes in Prometheus Text Format to byte stream >> interface >> > given to you. >> > - Pulsar histograms are exported in a way that is not conformant with >> > Prometheus, which means you can't get the p95 quantile on such >> histograms, >> > making them very hard to use in day to day life. >> >> What version of DataSketches is used to produce the histogram? Is is >> still an old Yahoo one, or are we using an updated one from Apache >> DataSketches? >> >> Seems like this is a single PR/small PIP for 3.1? > > > Histograms are a list of buckets, each is a counter. > Summary is a collection of values collected over a time window, which at > the end you get a calculation of the quantiles of those values: p95, p50, > and those are exported from Pulsar. > > Pulsar histogram do not use Data Sketches. They are just counters. > They are not adhere to Prometheus since: > a. The counter is expected to be cumulative, but Pulsar resets each bucket > counter to 0 every 1 min > b. The bucket upper range is expected to be written as an attribute "le" > but today it is encoded in the name of the metric itself. > > This is a breaking change, hence hard to mark in any small release. > This is why it's part of this PIP since so many things will break, and all > of them will break on a separate layer (OTel metrics), hence not break > anyone without their consent. > > > >> >> >> > - Too many metrics are rates which also delta reset every interval you >> > configure in Pulsar and restart, instead of relying on cumulative >> (ever >> > growing) counters and let Prometheus use its rate function. >> > - and many more issues >> > >> > From the developer perspective: >> > >> > - There are 4 different ways to define and record metrics in Pulsar: >> > Pulsar own metrics library, Prometheus Java Client, Bookkeeper metrics >> > library and plain native Java SDK objects (AtomicLong, ...). It's very >> > confusing for the developer and create inconsistencies for the end >> user >> > (e.g. Summary for example is different in each). >> > - Patching your metrics into "/metrics" Prometheus endpoint is >> > confusing, cumbersome and error prone. >> > - many more >> > >> > This proposal offers several key changes to solve that: >> > >> > - Cardinality (supporting 10k-100k topics per broker) is solved by >> > introducing a new aggregation level for metrics called Topic Metric >> Group. >> > Using configuration, you specify for each topic its group (using >> > wildcard/regex). This allows you to "zoom" out to a more detailed >> > granularity level like groups instead of namespaces, which you >> control how >> > many groups you'll have hence solving the cardinality issue, without >> > sacrificing level of detail too much. >> > - Fine-grained filtering mechanism, dynamic. You'll have rule-based >> > dynamic configuration, allowing you
Re: [DISCUSS] PIP-268: Add support of topic stats/stats-internal using client api
If I dive into the exact details of the cause of the performance implications in current Admin HTTP API: Do you think the root cause of the performance is the Jersey implementation of `AsyncResponse.resume(stats)` which takes a thread from a thread pool, serialize the object and then performs a blocking I/O write of the JSON string? If I compared that with Netty HTTP, it would write the same String using async I/O and not blocking a thread. Given the normally large response size of those objects, the response headers or request headers are negligible in terms of performance impact. In terms of accepting connection, both Netty and Jetty has async IO implementation. Compared Jetty and Jersey with Netty based binary TCP, if we end up writing a JSON string, the only difference I see is the blocking I/O of writing the response. WDYT? On Fri, May 12, 2023 at 7:29 AM Rajan Dhabalia wrote: > Communicating over binary protocol is more scalable and performant than > HTTP. Admin API over http has a long history of bottleneck and performance > issues which could also sometimes be a bottleneck for lookup requests and > that was the reason we introduced lookup over binary protocol as well. We > have multiple usecases which require fetching stats with relatively higher > rate and definitely we would like to avoid it over http which could be a > bottleneck for those applications or could be for others. > This PIP doesn't mention security so, let's not misinterpret the usecases. > Sometimes, pulsar is deployed behind the proxy and potentially used SNI > routing proxy which can't be used as http proxy and we would like to let > users access stats for a given topic using the same broker-service url > rather than having separate http endpoints. So, this api addresses scale, > performance, and use accessibility in pulsar. > > Thanks, > Rajan > > On Thu, May 11, 2023 at 6:24 AM Asaf Mesika wrote: > > > Before I dive into the PIP, I have several questions on the background > > provided below: > > > > > > On Tue, May 9, 2023 at 9:08 AM Rajan Dhabalia > > wrote: > > > > > Hi, > > > > > > Right now, Pulsar provides the topic's stats and stats-internal over > HTTP > > > admin API, and this stats data is used by user applications and also by > > > Pulsar internal components such as Pulsar-functions to derive the > certain > > > states of the applications. > > > for example, there are use cases where the application wants to check > the > > > topic's backlog, subscription's state (readPosition, list of > > > subscriptions), numberOfEntriesSinceFirstNotAckedMessage, etc to > > bootstrap > > > the application or handle the application’s resiliency and state > > > dynamically. Applications can retrieve this stats information by using > > the > > > broker’s admin HTTP APIs. > > > > > > However, stats retrieval over HTTP API doesn’t work well in use cases > > when > > > users would like to access this API at a higher scale when a large > number > > > of application nodes would like to use it over HTTP which could > overload > > > brokers and sometimes makes broker irresponsive and impact admin API > > > performance. It also becomes difficult when Pulsar is deployed in the > > cloud > > > behind the SNI proxy and applications also want to access large-scale > > stats > > > information periodically over different HTTP ports. Instead it would be > > > better if applications can fetch stats over on the same binary protocol > > for > > > scalability and accessibility reasons. > > > > > > > Why do you think using a binary protocol over HTTP would make more > > performant to respond to multiple calls at once? > > Same question but for the security issue - why do you think the HTTP port > > of admin API is harder to access than the binary protocol port? > > > > > > > > > > > > > > Therefore, there are multiple use cases where producer/consumer > > > applications need stats information for topics using the client library > > > over binary protocol. Hence, this PIP introduces client API for > producers > > > and consumers to access topic stats/internal-stats information which > can > > be > > > used by applications as needed. > > > > > > Please visit and review the PIP: > > > https://github.com/apache/pulsar/issues/20265 > > > > > > > > > Thanks, > > > > > > Rajan > > > > > >
Re: [DISCUSS] Replace stale bot with ping-pong workflow
Hi - I have not looked at all your links but I think this is a great idea. This will help everyone pay attention better. Best, Dave Sent from my iPhone > On May 14, 2023, at 12:33 AM, tison wrote: > > Of course, changing the workflow cannot magically increase the bandwidth to > handle stale issues. That is what the triage guide wants to encourage > committers to practice. But such a move can reduce the frustrating > experience and explicitly express who is responsible for taking the next > action to nudge the conversation. > > Best, > tison. > > > tison 于2023年5月14日周日 15:28写道： > >> Hi devs, >> >> Recently, I have handled a large number of stale issues and noticed that >> periodically notifying users that "the issue is stale" without any human >> reaction can be a frustrating experience, e.g., ISSUE-13925[1]. >> >> Learning from the INFRA JIRA project experience, I propose we replace the >> stale bot with a ping-pong workflow. That is - >> >> ping - Labeling waiting-for-reviewer on issue created and commented by >> non-committers >> pong - Labeling waiting-for-user on issue responded by committers >> >> Here is a demo implementation[2] you can refer to and you can try the >> workflow in my fork[3]. >> >> Previous references - >> >> * The triage guide[4] >> * [DISCUSS] Does stale bot make value for you?[5] >> * [COMMITTER ATTENTION] You can close stale issues as not planned [6] >> >> Looking forward to your feedback :D >> >> Best, >> tison. >> >> [1] https://github.com/apache/pulsar/issues/13925 >> [2] https://github.com/apache/pulsar/pull/20319 >> [3] https://github.com/tisonkun/pulsar >> [4] https://pulsar.apache.org/contribute/develop-triage >> [5] https://lists.apache.org/thread/tv774jqohdpx8x0dymsskrd90xwwfvgp >> [6] https://lists.apache.org/thread/x2c7xod8y0wvh14nsb6bknf0dq3r9gls >> >>
Re: [DISCUSS] Replace stale bot with ping-pong workflow
Of course, changing the workflow cannot magically increase the bandwidth to handle stale issues. That is what the triage guide wants to encourage committers to practice. But such a move can reduce the frustrating experience and explicitly express who is responsible for taking the next action to nudge the conversation. Best, tison. tison 于2023年5月14日周日 15:28写道： > Hi devs, > > Recently, I have handled a large number of stale issues and noticed that > periodically notifying users that "the issue is stale" without any human > reaction can be a frustrating experience, e.g., ISSUE-13925[1]. > > Learning from the INFRA JIRA project experience, I propose we replace the > stale bot with a ping-pong workflow. That is - > > ping - Labeling waiting-for-reviewer on issue created and commented by > non-committers > pong - Labeling waiting-for-user on issue responded by committers > > Here is a demo implementation[2] you can refer to and you can try the > workflow in my fork[3]. > > Previous references - > > * The triage guide[4] > * [DISCUSS] Does stale bot make value for you?[5] > * [COMMITTER ATTENTION] You can close stale issues as not planned [6] > > Looking forward to your feedback :D > > Best, > tison. > > [1] https://github.com/apache/pulsar/issues/13925 > [2] https://github.com/apache/pulsar/pull/20319 > [3] https://github.com/tisonkun/pulsar > [4] https://pulsar.apache.org/contribute/develop-triage > [5] https://lists.apache.org/thread/tv774jqohdpx8x0dymsskrd90xwwfvgp > [6] https://lists.apache.org/thread/x2c7xod8y0wvh14nsb6bknf0dq3r9gls > >
[DISCUSS] Replace stale bot with ping-pong workflow
Hi devs, Recently, I have handled a large number of stale issues and noticed that periodically notifying users that "the issue is stale" without any human reaction can be a frustrating experience, e.g., ISSUE-13925[1]. Learning from the INFRA JIRA project experience, I propose we replace the stale bot with a ping-pong workflow. That is - ping - Labeling waiting-for-reviewer on issue created and commented by non-committers pong - Labeling waiting-for-user on issue responded by committers Here is a demo implementation[2] you can refer to and you can try the workflow in my fork[3]. Previous references - * The triage guide[4] * [DISCUSS] Does stale bot make value for you?[5] * [COMMITTER ATTENTION] You can close stale issues as not planned [6] Looking forward to your feedback :D Best, tison. [1] https://github.com/apache/pulsar/issues/13925 [2] https://github.com/apache/pulsar/pull/20319 [3] https://github.com/tisonkun/pulsar [4] https://pulsar.apache.org/contribute/develop-triage [5] https://lists.apache.org/thread/tv774jqohdpx8x0dymsskrd90xwwfvgp [6] https://lists.apache.org/thread/x2c7xod8y0wvh14nsb6bknf0dq3r9gls
