Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
> `brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1` So, the "per-min" config can be more granular. Oh, okay. I thought we would end up only using one of `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` `brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec` ... these two. If there is per-sec configuration, we might simply have default per-second config value 60 (1minute). Thanks, JooHyukKim(vince) On Fri, Jul 7, 2023 at 2:09 PM Heesung Sohn wrote: > > but we should also consider per-minute could be too slow for some. > Could you clarify this part? > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60` is the same as > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1` So, the "per-min" > config can be more granular. > > Heesung > > On Thu, Jul 6, 2023 at 10:04 PM Joo Hyuk Kim wrote: > > > Hi, > > > > I agree that unloading per-sec might too fast, but we should also > consider > > per-minute could be too slow for some. > > So we might as well add configurability for unload rate (in seconds).But > I > > am concerend that we might end up having too many parameters 樂 > > > > Thanks, > > JooHyukKim(vince) > > > > On Fri, Jul 7, 2023 at 1:57 PM Heesung Sohn > > wrote: > > > > > Hi, > > > > > > I agree with configuring the > `shutdown.unloadNamespaceBundlesGracefully` > > > behavior by a dynamic config. > > > > > > Also, I wonder if the better config is > > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` to support a > > > slower rate. > > > One bundle unloading per sec could be too fast if there are many > > > topics(thousands) per bundle. > > > > > > Regards, > > > Heesung > > > > > > > > > On Thu, Jul 6, 2023 at 8:32 PM labuladong > > wrote: > > > > > > > HiJooHyukKim, > > > > > > > > > > > > That is what I want to do. Just use this function: > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 > > > > > > > > > > > > to replace this function in closeAsync: > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > > > > > > > > > > Now admin API support to limit the unloadconcurrency, but > > graceful > > > > shutdown doesn't support, it's easy to improve. > > > > > > > > > > > > In addition, I suggest storing the `maxConcurrentUnloadPerSec` as a > > > broker > > > > dynamic config, because users may want to change this value in > > different > > > > situation. What do you think, static config or dynamic config? > > > > > On Fri, Jul 7, 2023 at 2:09 PM Heesung Sohn wrote: > > but we should also consider per-minute could be too slow for some. > Could you clarify this part? > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60` is the same as > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1` So, the "per-min" > config can be more granular. > > Heesung > > On Thu, Jul 6, 2023 at 10:04 PM Joo Hyuk Kim wrote: > > > Hi, > > > > I agree that unloading per-sec might too fast, but we should also > consider > > per-minute could be too slow for some. > > So we might as well add configurability for unload rate (in seconds).But > I > > am concerend that we might end up having too many parameters 樂 > > > > Thanks, > > JooHyukKim(vince) > > > > On Fri, Jul 7, 2023 at 1:57 PM Heesung Sohn > > wrote: > > > > > Hi, > > > > > > I agree with configuring the > `shutdown.unloadNamespaceBundlesGracefully` > > > behavior by a dynamic config. > > > > > > Also, I wonder if the better config is > > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` to support a > > > slower rate. > > > One bundle unloading per sec could be too fast if there are many > > > topics(thousands) per bundle. > > > > > > Regards, > > > Heesung > > > > > > > > > On Thu, Jul 6, 2023 at 8:32 PM labuladong > > wrote: > > > > > > > HiJooHyukKim, > > > > > > > > > > > > That is what I want to do. Just use this function: > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 > > > > > > > > > > > > to replace this function in closeAsync: > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > > > > > > > > > > Now admin API support to limit the unloadconcurrency, but > > graceful > > > > shutdown doesn't support, it's easy to improve. > > > > > > > > > > > > In addition, I suggest storing the `maxConcurrentUnloadPerSec` as a > > > broker > > > > dynamic config, because users may want to change this value in > > different > > > > situation. What do you think, static
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
> but we should also consider per-minute could be too slow for some. Could you clarify this part? `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60` is the same as `brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1` So, the "per-min" config can be more granular. Heesung On Thu, Jul 6, 2023 at 10:04 PM Joo Hyuk Kim wrote: > Hi, > > I agree that unloading per-sec might too fast, but we should also consider > per-minute could be too slow for some. > So we might as well add configurability for unload rate (in seconds).But I > am concerend that we might end up having too many parameters 樂 > > Thanks, > JooHyukKim(vince) > > On Fri, Jul 7, 2023 at 1:57 PM Heesung Sohn > wrote: > > > Hi, > > > > I agree with configuring the `shutdown.unloadNamespaceBundlesGracefully` > > behavior by a dynamic config. > > > > Also, I wonder if the better config is > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` to support a > > slower rate. > > One bundle unloading per sec could be too fast if there are many > > topics(thousands) per bundle. > > > > Regards, > > Heesung > > > > > > On Thu, Jul 6, 2023 at 8:32 PM labuladong > wrote: > > > > > HiJooHyukKim, > > > > > > > > > That is what I want to do. Just use this function: > > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 > > > > > > > > > to replace this function in closeAsync: > > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > > > > > > > Now admin API support to limit the unloadconcurrency, but > graceful > > > shutdown doesn't support, it's easy to improve. > > > > > > > > > In addition, I suggest storing the `maxConcurrentUnloadPerSec` as a > > broker > > > dynamic config, because users may want to change this value in > different > > > situation. What do you think, static config or dynamic config? > > >
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
Hi, I agree that unloading per-sec might too fast, but we should also consider per-minute could be too slow for some. So we might as well add configurability for unload rate (in seconds).But I am concerend that we might end up having too many parameters 樂 Thanks, JooHyukKim(vince) On Fri, Jul 7, 2023 at 1:57 PM Heesung Sohn wrote: > Hi, > > I agree with configuring the `shutdown.unloadNamespaceBundlesGracefully` > behavior by a dynamic config. > > Also, I wonder if the better config is > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` to support a > slower rate. > One bundle unloading per sec could be too fast if there are many > topics(thousands) per bundle. > > Regards, > Heesung > > > On Thu, Jul 6, 2023 at 8:32 PM labuladong wrote: > > > HiJooHyukKim, > > > > > > That is what I want to do. Just use this function: > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 > > > > > > to replace this function in closeAsync: > > > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > > > > Now admin API support to limit the unloadconcurrency, but graceful > > shutdown doesn't support, it's easy to improve. > > > > > > In addition, I suggest storing the `maxConcurrentUnloadPerSec` as a > broker > > dynamic config, because users may want to change this value in different > > situation. What do you think, static config or dynamic config? >
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
Hi, I agree with configuring the `shutdown.unloadNamespaceBundlesGracefully` behavior by a dynamic config. Also, I wonder if the better config is `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` to support a slower rate. One bundle unloading per sec could be too fast if there are many topics(thousands) per bundle. Regards, Heesung On Thu, Jul 6, 2023 at 8:32 PM labuladong wrote: > HiJooHyukKim, > > > That is what I want to do. Just use this function: > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 > > > to replace this function in closeAsync: > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > Now admin API support to limit the unloadconcurrency, but graceful > shutdown doesn't support, it's easy to improve. > > > In addition, I suggest storing the `maxConcurrentUnloadPerSec` as a broker > dynamic config, because users may want to change this value in different > situation. What do you think, static config or dynamic config?
Re: [VOTE] PIP-273: Enable hostname verification by default
Closing vote as passed with 3 binding +1s (Michael, Yunze, and Enrico) and 1 non-binding +1 (Zixuan). I'll make sure the PRs are up to date and will start requesting reviews to get them merged. Thanks, Michael On Tue, Jun 6, 2023 at 8:54 AM Zixuan Liu wrote: > > +1 (non-binding) > > Thanks, > Zixuan > > Enrico Olivelli 于2023年6月6日周二 16:40写道： > > > > +1 (binding) > > > > Enrico > > > > Il Mar 6 Giu 2023, 10:32 Yunze Xu ha scritto: > > > > > +1 (binding) > > > > > > Thanks, > > > Yunze > > > > > > On Tue, Jun 6, 2023 at 2:09 PM Michael Marshall > > > wrote: > > > > > > > > +1 (binding) > > > > > > > > - Michael > > > > > > > > On Tue, Jun 6, 2023 at 1:08 AM Michael Marshall > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > > This is the vote thread for PIP 273. > > > > > > > > > > PR: https://github.com/apache/pulsar/pull/20453 > > > > > Discussion: > > > https://lists.apache.org/thread/7dprl8wvq4r2y8n4p32gwrl6jn97yckq > > > > > > > > > > The vote will be open for at least 48 hours. > > > > > > > > > > Thanks, > > > > > Michael > > >
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
Thanks, JooHyukKim (vince) On Fri, Jul 7, 2023 at 11:59 AM Joo Hyuk Kim wrote: > > When Pulsar undergoes a graceful shutdown, it unloads all topics, but > there is no limit on the rate of unloading > > Is it possible that following code might already cover what we need here? > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 > > On Fri, Jul 7, 2023 at 11:06 AM Dave Fisher wrote: > >> Hi - >> >> Interesting. I’m curious to know what Heesung, Lari, and others think. >> >> Best, >> Dave >> >> Sent from my iPhone >> >> > On Jul 6, 2023, at 7:01 PM, labuladong wrote: >> > >> > Dear Apache Pulsar community, >> > >> > >> > When Pulsar undergoes a graceful shutdown, it unloads all topics, but >> there is no limit on the rate of unloading. As a result, during a rolling >> upgrade, the broker cluster might generate a large volume of unload >> requests in a short period, which could affect the service of the broker. >> Implementing a limit on the unload rate would make the upgrade process >> smoother. >> > >> > >> > This issue has been previously raised in a pull request and a solution >> was proposed by adding an unload interface with concurrency control: >> > >> > >> > https://github.com/apache/pulsar/pull/14114 >> > >> > >> > However, the current broker graceful shutdown doesn't use this >> interface. Hence, my proposal is pretty straightforward: >> > >> > >> > 1. Add a dynamic config to the Pulsar broker to control the concurrency >> of unloading. >> > >> > >> > 2. Modify the following code to use the function with the concurrency >> parameter for unloading. >> > >> > >> > >> https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 >> > >> > >> > I look forward to your thoughts on this. >> > >> > >> > >> > Best regards, donglai >> >>
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
> When Pulsar undergoes a graceful shutdown, it unloads all topics, but there is no limit on the rate of unloading Is it possible that following code might already cover what we need here? https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557 On Fri, Jul 7, 2023 at 11:06 AM Dave Fisher wrote: > Hi - > > Interesting. I’m curious to know what Heesung, Lari, and others think. > > Best, > Dave > > Sent from my iPhone > > > On Jul 6, 2023, at 7:01 PM, labuladong wrote: > > > > Dear Apache Pulsar community, > > > > > > When Pulsar undergoes a graceful shutdown, it unloads all topics, but > there is no limit on the rate of unloading. As a result, during a rolling > upgrade, the broker cluster might generate a large volume of unload > requests in a short period, which could affect the service of the broker. > Implementing a limit on the unload rate would make the upgrade process > smoother. > > > > > > This issue has been previously raised in a pull request and a solution > was proposed by adding an unload interface with concurrency control: > > > > > > https://github.com/apache/pulsar/pull/14114 > > > > > > However, the current broker graceful shutdown doesn't use this > interface. Hence, my proposal is pretty straightforward: > > > > > > 1. Add a dynamic config to the Pulsar broker to control the concurrency > of unloading. > > > > > > 2. Modify the following code to use the function with the concurrency > parameter for unloading. > > > > > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > > > > I look forward to your thoughts on this. > > > > > > > > Best regards, donglai > >
RE: [VOTE] Pulsar Node.js Client Release 1.9.0 Candidate 1
+1 (binding) - verified checksum and signature - confirmed that the build was successful - ran produce/consume Regards, Masahiro Sakamoto Yahoo Japan Corp. E-mail: massa...@yahoo-corp.jp
Re: Request to contribute to Apache Pulsar JIRA maintenance and documentation
Adding the OP who is not subscribed. Sent from my iPhone > On Jul 6, 2023, at 6:35 PM, Yu wrote: > > Hi Yuvaraj, > > Thank you for your willingness to contribute to the Pulsar documentation! > > Feel free to select your desired issues from > https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Adoc-required > . > > You can find the documentation contribution guide at > https://pulsar.apache.org/contribute/document-intro/, and don't hesitate to > reach out to me for any assistance or to have your pull requests reviewed. > > Yu > > >> On Fri, Jul 7, 2023 at 8:50 AM Yuvaraj Madeshwaran >> wrote: >> >> Dear Apache Pulsar Foundation, >> >> I am writing to express my interest in contributing to the JIRA dashboard >> maintenance and documentation of Apache Pulsar Github repos. I have >> developed a deep understanding of the software and JIRA. I am also >> proficient in writing technical documentation. >> >> I believe that my skills and experience would be a valuable asset to the >> Apache Pulsar project. I am a highly motivated and detail-oriented >> individual, and I am always willing to go the extra mile. I am also a quick >> learner, and I am always eager to learn new things. >> >> I am particularly interested in contributing to the following areas: >> JIRA board maintenance >> Documentation >> Testing >> >> I am available to contribute on a part-time basis. I have attached my >> resume for your review. I have also included a link to my GitHub profile >> https://github.com/yuvarajmms . >> >> Thank you for your time and consideration. I look forward to hearing from >> you soon. >> >> Sincerely, >> Yuvaraj Madheswaran >>
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
Hi - Interesting. I’m curious to know what Heesung, Lari, and others think. Best, Dave Sent from my iPhone > On Jul 6, 2023, at 7:01 PM, labuladong wrote: > > Dear Apache Pulsar community, > > > When Pulsar undergoes a graceful shutdown, it unloads all topics, but there > is no limit on the rate of unloading. As a result, during a rolling upgrade, > the broker cluster might generate a large volume of unload requests in a > short period, which could affect the service of the broker. Implementing a > limit on the unload rate would make the upgrade process smoother. > > > This issue has been previously raised in a pull request and a solution was > proposed by adding an unload interface with concurrency control: > > > https://github.com/apache/pulsar/pull/14114 > > > However, the current broker graceful shutdown doesn't use this interface. > Hence, my proposal is pretty straightforward: > > > 1. Add a dynamic config to the Pulsar broker to control the concurrency of > unloading. > > > 2. Modify the following code to use the function with the concurrency > parameter for unloading. > > > https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791 > > > I look forward to your thoughts on this. > > > > Best regards, donglai
Re: Request to contribute to Apache Pulsar JIRA maintenance and documentation
Hi Yuvaraj, Thank you for your willingness to contribute to the Pulsar documentation! Feel free to select your desired issues from https://github.com/apache/pulsar/issues?q=is%3Aopen+is%3Aissue+label%3Adoc-required . You can find the documentation contribution guide at https://pulsar.apache.org/contribute/document-intro/, and don't hesitate to reach out to me for any assistance or to have your pull requests reviewed. Yu On Fri, Jul 7, 2023 at 8:50 AM Yuvaraj Madeshwaran wrote: > Dear Apache Pulsar Foundation, > > I am writing to express my interest in contributing to the JIRA dashboard > maintenance and documentation of Apache Pulsar Github repos. I have > developed a deep understanding of the software and JIRA. I am also > proficient in writing technical documentation. > > I believe that my skills and experience would be a valuable asset to the > Apache Pulsar project. I am a highly motivated and detail-oriented > individual, and I am always willing to go the extra mile. I am also a quick > learner, and I am always eager to learn new things. > > I am particularly interested in contributing to the following areas: > JIRA board maintenance > Documentation > Testing > > I am available to contribute on a part-time basis. I have attached my > resume for your review. I have also included a link to my GitHub profile > https://github.com/yuvarajmms . > > Thank you for your time and consideration. I look forward to hearing from > you soon. > > Sincerely, > Yuvaraj Madheswaran >
[DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager
Hi dev, I proposed this PIP, https://github.com/apache/pulsar/pull/20748, to make unloaded clients directly and gracefully connect to new owner brokers without lookups. Please take a look and let me know what you think. Regards, Heesung
Re: [VOTE] Pulsar Node.js Client Release 1.9.0 Candidate 1
+1 (non-binding) - Verified checksum and signature - Build from source codes - Verified pub/sub Hideaki Oguni Yahoo Japan Corp.
Re: [VOTE] PIP-276: Add metric `pulsar_topic_load_times
Thank you all. We've added the doc in https://github.com/apache/pulsar-site/pull/631 On Mon, Jul 3, 2023 at 11:15 AM guo jiwei wrote: > Close the vote with 4(+1 binding) 2(+1 non-binding) 0(-1) > > binding: > Hang > Yunze > Mattison > Penghui > > non-binding > Yubiao > Asaf > > > Regards > Jiwei Guo (Tboy) > > > On Mon, Jun 26, 2023 at 10:30 PM Hang Chen wrote: > > > +1 (binding) > > > > Thanks, > > Hang > > > > Yunze Xu 于2023年6月26日周一 19:59写道： > > > > > > +1 (binding) > > > > > > Thanks, > > > Yunze > > > > > > > > > > > > > > > > On Jun 20, 2023, at 10:53, mattisonc...@gmail.com wrote: > > > > > > > > +1(binding) > > > > > > > > Best, > > > > Mattison > > > > On 20 Jun 2023 at 10:45 +0800, PengHui Li , > wrote: > > > >> +1 (binding) > > > >> > > > >> Thanks, > > > >> Penghui > > > >> > > > >> On Tue, Jun 20, 2023 at 10:40 AM Yubiao Feng > > > >> wrote: > > > >> > > > >>> Voting +1 (non-binding) > > > >>> > > > >>> Thanks > > > >>> Yubiao Feng > > > >>> > > > >>> On Mon, Jun 19, 2023 at 5:21 PM Asaf Mesika > > > wrote: > > > >>> > > > > Voting +1 (non-binding) > > > > > > > > On Fri, Jun 16, 2023 at 12:23 PM guo jiwei > > > wrote: > > > > > > > >>> @Asaf Thanks, I have addressed the comment. > > > >>> > > > >>> Regards > > > >>> Jiwei Guo (Tboy) > > > >>> > > > >>> > > > >>> On Fri, Jun 16, 2023 at 3:55 AM Asaf Mesika < > > asaf.mes...@gmail.com> > > > > wrote: > > > >>> > > > > -1 (non-binding) > > > > > > > > I'm perfectly ok with the idea; just please fix the document. > > It > > > >>> looks > > > >>> too > > > > messy. Even 1 paragraph changes can look neat and clean. > > > > I left notes in the draft PR you opened for the pip. > > > > > > > > I'll change my non-binding vote once that's done. > > > > > > > > On Thu, Jun 15, 2023 at 11:07 AM guo jiwei < > > techno...@apache.org> > > > > wrote: > > > > > > > >>> Hi, community: > > > >>> The metrics are all started with `pulsar_`, so that both > > users > > > > and > > > >>> operators can quickly find the metrics of the entire system > > through > > > >>> this prefix. However, due to some other reasons, it was > > found that > > > >>> `topic_load_times` was missing the prefix, so want to get > it > > right. > > > >>> In the master branch : > > > >>> * `pulsar_topic_load_times`: Add this new metric which has > > the > > > >>> same > > > >>> meaning as `topic_load_times` > > > >>> * `topic_load_times`: Mark this metric as deprecated and > > > >>> remove > > > >>> it > > > > in > > > >>> the next version > > > >>> > > > >>> PIP: https://github.com/apache/pulsar/pull/20518 > > > >>> > > > >>> Regards > > > >>> Jiwei Guo (Tboy) > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > >
