Re: [DISCUSS] PIP-285: Add pulsar_subscription_back_log_duration metric
+1 houxiaoyu 于2023年7月24日周一 20:00写道： > +1 > > Thanks > Xiaoyu Hou > > 丛搏 于2023年7月24日周一 19:58写道： > > > Hi, Pulsar Community > > > > I opened a new PIP design PR. > > > > https://github.com/apache/pulsar/pull/20859 > > > > Thanks, > > Bo > > > -- BR, Qiang Huang
Re: [ANNOUNCE] Nicolò Boschi as new PMC member in Apache Pulsar
Congratulations !!! Zike Yang 于2023年1月30日周一 16:41写道： > Congratulations! > > BR, > Zike Yang > > On Mon, Jan 30, 2023 at 3:16 PM houxiaoyu wrote: > > > > Congratulations ! > > > > Best > > Xiaoyu Hou > > > > Max Xu 于2023年1月30日周一 12:34写道： > > > > > Congratulations! Nicolò > > > > > > Best, > > > Max Xu > > > > > > > > > On Fri, Jan 20, 2023 at 8:36 PM Lari Hotari > wrote: > > > > > > > Dear Community, > > > > > > > > We are thrilled to announce that Nicolò Boschi > > > > (https://github.com/nicoloboschi) has been invited and has accepted > the > > > > role of member of the Apache Pulsar Project Management Committee > (PMC). > > > > > > > > Nicolò Boschi has been a vital asset to our community, consistently > > > > demonstrating his dedication and active participation through > > > > significant contributions such as the development of the Pulsar Shell > > > > and numerous bug fixes, security enhancements, and improvements to > > > > Pulsar and its CI pipeline. In addition to his technical > contributions, > > > > Nicolò also plays an important role in reviewing pull requests and > > > > ensuring the overall quality of our project. We look forward to his > > > > continued contributions. > > > > > > > > On behalf of the Pulsar PMC, we extend a warm welcome and > > > > congratulations to Nicolò Boschi. > > > > > > > > Best regards, Lari, on behalf of the Pulsar PMC > > > > > > > > -- BR, Qiang Huang
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congrats!!! Max Xu 于2023年1月30日周一 12:33写道： > Congratulations! Bo > > Best, > Max Xu > > > On Wed, Jan 18, 2023 at 9:50 PM PengHui Li wrote: > > > Hi all, > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > > (https://github.com/congbobo184) as a member of the PMC and we are > > pleased to announce that he has accepted. > > > > He is very active in the community in the past few years and made a lot > of > > great contributions > > such as transactions and schemas. > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > Best Regards, > > Penghui on behalf of the Pulsar PMC > > > -- BR, Qiang Huang
Re: [ANNOUNCE] New Committer: Baodi Shi
Congratulations! Max Xu 于2023年1月30日周一 12:34写道： > Congratulations! Baodi > > Best, > Max Xu > > > On Wed, Jan 18, 2023 at 9:36 PM Yunze Xu wrote: > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > Baodi Shi (https://github.com/shibd) to become a committer and we are > > pleased to announce that he has accepted. > > > > Being a committer enables easier contribution to the project since > > there is no need to go via the patch submission process. This should > > enable better productivity. > > > > Welcome and congratulations, Baodi Shi! > > > > Please join us in congratulating and welcoming Baodi Shi onboard! > > > > Thanks, > > Yunze on behalf of the Pulsar PMC > > > -- BR, Qiang Huang
Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar
Congratulations!!! Yunze Jun Ma 于2022年12月31日周六 17:26写道： > Congrats and well deserved! > > From: Dave Fisher > Sent: Saturday, December 31, 2022 1:52 > To: dev@pulsar.apache.org > Subject: Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar > > Congratulations Yunze! Well deserved! > > All the best, > Dave > > > > On Dec 29, 2022, at 4:42 AM, Haiting Jiang > wrote: > > > > Hi all, > > > > The Apache Pulsar Project Management Committee (PMC) has invited Yunze Xu > > (https://github.com/BewareMyPower) as a member of the PMC and we are > > pleased to announce that he has accepted. > > > > He is very active in the community in the past few years and made a lot > of great contributions. > > > > Welcome Yunze to the Apache Pulsar PMC. > > > > Best Regards, > > Haiting Jiang on behalf of the Pulsar PMC > > -- BR, Qiang Huang
Re: [ANNOUNCE] New Committer: Cong Zhao
Congrats! Nicolò Boschi 于2022年11月22日周二 15:22写道： > Congrats! > Nicolò Boschi > > > Il giorno mar 22 nov 2022 alle ore 06:23 Hang Chen > ha > scritto: > > > Congrats! > > > > Best, > > Hang > > > > tison 于2022年11月22日周二 13:15写道： > > > > > > Congrats! > > > > > > Best, > > > tison. > > > > > > > > > Max Xu 于2022年11月22日周二 13:14写道： > > > > > > > Congratulations! Cong > > > > > > > > Best, > > > > Max Xu > > > > > > > > > > > > On Mon, Nov 21, 2022 at 12:10 PM Haiting Jiang < > > jianghait...@apache.org> > > > > wrote: > > > > > > > > > The Project Management Committee (PMC) for Apache Pulsar has > invited > > > > > Cong Zhao (https://github.com/coderzc) > > > > > to become a committer and we are pleased to announce that he has > > > > accepted. > > > > > > > > > > Being a committer enables easier contribution to the > > > > > project since there is no need to go via the patch > > > > > submission process. This should enable better productivity. > > > > > > > > > > Welcome and congratulations, Cong Zhao! > > > > > > > > > > Please join us in congratulating and welcoming Cong Zhao onboard! > > > > > > > > > > Best Regards, > > > > > Haiting on behalf of the Pulsar PMC > > > > > > > > > > > > -- BR, Qiang Huang
Re: [ANNOUNCE] New Committer: Zili Chen
Congratulations! Zike Yang 于2022年11月14日周一 13:45写道： > Hi, tison, > > Congratulations and welcome! > > BR, > Zike Yang > > On Mon, Nov 14, 2022 at 12:58 PM Kai Wang > wrote: > > > > Congratulations! tison > > > > Thanks, > > Kai > > On Nov 10, 2022 at 8:16 AM +0800, dev@pulsar.apache.org, wrote: > > > > > > Congratulations! tison > -- BR, Qiang Huang
Re: [ANNOUNCE] New Committer: Lin Chen
Congratulations! houxiaoyu 于2022年11月14日周一 13:40写道： > Congrats! > > Best, > Xiaoyu Hou > > PengHui Li 于2022年11月14日周一 12:28写道： > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > Lin Chen (https://github.com/lordcheng10) > > to become a committer and we are pleased to announce that he has > accepted. > > > > Being a committer enables easier contribution to the > > project since there is no need to go via the patch > > submission process. This should enable better productivity. > > > > Welcome and congratulations, Lin Chen! > > > > Please join us in congratulating and welcoming Lin Chen onboard! > > > > Best Regards, > > Penghui on behalf of the Pulsar PMC > > > -- BR, Qiang Huang
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
+1 Enrico Olivelli 于2022年10月25日周二 22:32写道： > +1 > > awesome work > > Enrico > > Il giorno mar 25 ott 2022 alle ore 16:28 PengHui Li > ha scritto: > > > > +1 > > > > Penghui > > > > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 wrote: > > > > > Hi, pulsar community: > > > > > > I start a PIP about `User-friendly acknowledgeCumulative API on a > > > partitioned topic or multi-topics` > > > > > > discuss thread: > > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131 > > > > > > PIP: https://github.com/apache/pulsar/issues/18182 > > > > > > Thanks, > > > bo > > > > -- BR, Qiang Huang
Re: [ANNOUNCE] Haiting Jiang as a new PMC member in Apache Pulsar
Congratulations Haiting !!! Zixuan Liu 于2022年10月18日周二 23:46写道： > Congrats! > > tison 于2022年10月18日周二 23:08写道： > > > Congrats! > > > > Best, > > tison. > > > > > > Nicolò Boschi 于2022年10月18日周二 23:00写道： > > > > > Congrats!! > > > > > > Nicolò Boschi > > > > > > > > > Il giorno mar 18 ott 2022 alle ore 16:55 Michael Marshall < > > > mmarsh...@apache.org> ha scritto: > > > > > > > Congratulations Haiting! > > > > > > > > - Michael > > > > > > > > On Tue, Oct 18, 2022 at 3:06 AM Hang Chen > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited > > > Haiting > > > > Jiang > > > > > (https://github.com/Jason918) as a member of the PMC and we are > > > > > pleased to announce that he has accepted. > > > > > > > > > > He is very active in the community on GitHub. > > > > > > > > > > Welcome Haiting to the Apache Pulsar PMC > > > > > > > > > > Best Regards, > > > > > Hang Chen on behalf of the Pulsar PMC > > > > > > > > > > -- BR, Qiang Huang
Re: [VOTE] Pulsar Release 2.10.2 Candidate 3
+1(non-binding) - Checked signatures/checksums - Built from sources using `mvn clean install -DskipTests` using JDK 17 - Run the standalone - Validate Connectors - Validate Pub/Sub and Java Functions - Validate Stateful Functions Michael Marshall 于2022年10月17日周一 12:31写道： > +1 (binding) > > - Verified signatures for 48 artifacts > - Verified checksums for 48 artifacts > - Built from sources using `mvn clean install -DskipTests` using > Temurin-11.0.14+9 JDK > - Verified `mvn apache-rat:check` passes > - Ran pulsar standalone, verified some pulsar-admin commands, verified > perf produce/consume worked using JDK 11 > > Thanks for running the release Haiting! > > - Michael > > On Sun, Oct 16, 2022 at 10:40 PM guo jiwei wrote: > > > > +1 (binding) > > > > - Build from sources > > - Checksum and signatures > > - Run Pulsar standalone and produce/consume case > > - Validate Pub/Sub and Java Functions > > - Validate Connectors > > - Validate Stateful Functions > > > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Thu, Oct 13, 2022 at 10:52 PM Enrico Olivelli > > wrote: > > > > > +1 (binding) > > > - validated RAT, checksums and signatures > > > - built from sources and run smoke tests on the built binaries (JDK11 > > > temurin on Mac) > > > > > > Thanks > > > Enrico > > > > > > Il giorno gio 13 ott 2022 alle ore 13:10 houxiaoyu > > > ha scritto: > > > > > > > > +1 (non-binding) > > > > > > > > - Checksum and signatures > > > > - Build on Mac, using JDK17 > > > > - Run Pulsar standalone and produce/consume case > > > > > > > > Thanks, > > > > Xiaoyu Hou > > > > > > > > Haiting Jiang 于2022年10月13日周四 18:52写道： > > > > > > > > > This is the third release candidate for Apache Pulsar, version > 2.10.2. > > > > > > > > > > This release contains 252 commits by 57 contributors. > > > > > > https://github.com/apache/pulsar/compare/v2.10.1...v2.10.2-candidate-3 > > > > > > > > > > CI for this release candidate > > > > > https://github.com/Jason918/pulsar/pull/11 > > > > > > > > > > *** Please download, test and vote on this release. This vote will > stay > > > > > open > > > > > for at least 72 hours *** > > > > > > > > > > Note that we are voting upon the source (tag), binaries are > provided > > > for > > > > > convenience. > > > > > > > > > > Source and binary files: > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.2-candidate-3/ > > > > > > > > > > SHA-512 checksums: > > > > > > > > > > > > > > c136ef4f47b3b4edfb99d8a927ba70df19c12ac28d1711825bda4ef09e543d3d473ce732996244042ec3387f0e3cafdde6c8f5a0d7d18ad24429e781e65d3328 > > > > > ./apache-pulsar-2.10.2-bin.tar.gz > > > > > > > > > > > > > > 1fd10fbb48e734dbb95f20b0d396cb9f875a0a1c93f44b639dc730379ce93bc59ce20bfba137f28897c1e5b3351b08ca3341863c903cf2b6d3dc7108046f2f8f > > > > > ./apache-pulsar-2.10.2-src.tar.gz > > > > > > > > > > Maven staging repo: > > > > > > > > > https://repository.apache.org/content/repositories/orgapachepulsar-1187/ > > > > > > > > > > The tag to be voted upon: > > > > > v2.10.2-candidate-3 (11b5df797b2e9cb48dfc38137f0b7ef736a8a47c) > > > > > https://github.com/apache/pulsar/releases/tag/v2.10.2-candidate-3 > > > > > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > > > > > Docker images: > > > > > > > > > > > > > > https://hub.docker.com/layers/jason918/pulsar/2.10.2/images/sha256-6a501a0dc0e05b6bbd76e0f9edd1ff7223d0447bd318332dcf9551b48e831cf9 > > > > > > > > > > > > > > https://hub.docker.com/layers/jason918/pulsar-all/2.10.2/images/sha256-fd3e027771f08eb224a3a9e8874514f0f258a0356059120d3f25f151f96506c8 > > > > > > > > > > Please download the source package, and follow the > > > > > release-candidate-validation doc to build > > > > > and run the Pulsar standalone service. > > > > > > > > > > > > > > https://github.com/apache/pulsar/blob/master/wiki/release/release-candidate-validation.md > > > > > > > > > > Thanks, > > > > > Haiting > > > > > > > > > -- BR, Qiang Huang
Re: [VOTE] Pulsar Client Go Release 0.9.0 Candidate 2
+1 (non-binding) - Build from the source code - Run producer and consumer guo jiwei 于2022年10月8日周六 11:27写道： > +1 (binding) > > - Build from the source code > - Run producer and consumer > > > Regards > Jiwei Guo (Tboy) > > > On Sat, Oct 8, 2022 at 11:17 AM Zike Yang wrote: > > > +1 (non-binding) > > - Build from the source code > > - Run producer and consumer > > - Run pulsar-perf produce and consume > > > > Thanks, > > Zike Yang > > > > On Sat, Oct 8, 2022 at 10:53 AM Hang Chen wrote: > > > > > > +1(binding) > > > - Check sha 512 for the source code > > > - Build from the source code > > > - Run produce and consume > > > > > > Thanks, > > > Hang > > > > > > PengHui Li 于2022年9月29日周四 22:06写道： > > > > > > > > +1 (binding) > > > > > > > > - build from the source code > > > > - test the produce and consume > > > > > > > > Penghui > > > > > > > > On Thu, Sep 29, 2022 at 2:43 PM Guangning E > > wrote: > > > > > > > > > +1(non-binding) > > > > > - Check sha 512 value for source code > > > > > - Check go build producer example > > > > > > > > > > Thanks, > > > > > Guangning > > > > > > > > > > r...@apache.org 于2022年9月29日周四 14:35写道： > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Please review and vote on the release candidate #2 for the > version > > 0.9.0, > > > > > > as follows: > > > > > > [ ] +1, Approve the release > > > > > > [ ] -1, Do not approve the release (please provide specific > > comments) > > > > > > > > > > > > This is the second release candidate for Apache Pulsar Go client, > > > > > > version 0.9.0. > > > > > > > > > > > > It fixes the following issues: > > > > > > https://github.com/apache/pulsar-client-go/milestone/10?closed=1 > > > > > > > > > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign > this > > > > > > release: > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > > > > > > > Please download these packages and review this release candidate: > > > > > > - Review release notes > > > > > https://github.com/apache/pulsar-client-go/pull/804 > > > > > > - Download the source package (verify shasum, and asc) and follow > > the > > > > > > README.md to build and run the pulsar-client-go. > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > majority > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > Source file: > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.9.0-candidate-2/ > > > > > > > > > > > > The tag to be voted upon: > > > > > > v0.9.0 > > > > > > > > > > > > > > https://github.com/apache/pulsar-client-go/releases/tag/v0.9.0-candidate-2 > > > > > > > > > > > > SHA-512 checksums: > > > > > > > > > > > > > > > > > > > > 9731d6a0615288e77feb4b73fedbbdf6d275ebefeee3cee5fc4e849f38789863f0532c7e8b93eb1e601bd98f9bb21d50a714fcf87fac9987a745a052bbe23ca3 > > > > > > apache-pulsar-client-go-0.9.0-candidate-2-src.tar.gz > > > > > > > > > > > > Best, > > > > > > > > > > > > Xiaolong Ran > > > > > > > > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] PIP-209: Separate C++/Python clients to own repositories
+1 Zixuan Liu 于2022年9月26日周一 23:42写道： > +1 > > Thanks, > Zixuan > > Anon Hxy 于2022年9月26日周一 17:52写道： > > > +1 LGTM > > > > Thanks, > > Xiaoyu Hou > > > > Enrico Olivelli 于2022年9月26日周一 17:39写道： > > > > > +100 to this ! > > > > > > Thanks for this proposal > > > > > > Enrico > > > > > > Il giorno lun 26 set 2022 alle ore 10:04 Zike Yang > ha > > > scritto: > > > > > > > > +1. Looks good to me. > > > > > > > > Do we need to move the `Apache Pulsar / Multi Clients` Github project > > > > to the corresponding repos? > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > Zike Yang > > > > > > > > > > > > On Fri, Sep 23, 2022 at 7:22 AM Matteo Merli > > > > wrote: > > > > > > > > > > -- > > > > > Matteo Merli > > > > > > > > > > > > > > > On Tue, Sep 20, 2022 at 8:14 PM Michael Marshall < > > mmarsh...@apache.org> > > > wrote: > > > > > > > > > > > > Great proposal, thanks Matteo. > > > > > > > > > > > > I think I agree with splitting out the client into two repos. One > > > > > > issue is that new C++ features will lag in the python client > > because > > > > > > the C++ client will first need to be released. Quick releases > will > > > > > > likely help there. > > > > > > > > > > Yes, decoupling Java, C++ and Python releases will make each of > them > > > > > much easier. > > > > > > > > > > We'll be able to do patch releases with a tenth of the manual work > > > involved. > > > > > > > > > > > > The client <--> broker compatibility is in general always > > > guaranteed. > > > > > > > > > > > > I think we should make this more visible in our Pulsar > > documentation. > > > > > > It's a fantastic feature, and I get the sense that it is not well > > > > > > known. > > > > > > > > > > Agree, it's something that still surprises a lot of people. > > > > > > -- BR, Qiang Huang
Re: [VOTE] PIP-209: Separate C++/Python clients to own repositories
+1 (non-binding) Yunze Xu 于2022年9月27日周二 11:35写道： > +1 (non-binding) > > Thanks, > Yunze > > > > > > On Sep 27, 2022, at 11:11, Zixuan Liu wrote: > > > > +1(non-binding) > > > > Thanks, > > Zixuan > > > > Haiting Jiang 于2022年9月27日周二 09:56写道： > > > >> +1 (non) > >> > >> Haiting > >> > >> On Tue, Sep 27, 2022 at 9:30 AM Zike Yang wrote: > >>> > >>> +1 (non-binding) > >>> > >>> > >>> Zike Yang > >>> > >>> On Tue, Sep 27, 2022 at 9:24 AM Kai Wang > >> wrote: > >>>> > >>>> +1 (non-binding) > >>>> > >>>> Thanks, > >>>> Kai > >> > > -- BR, Qiang Huang
Re: [VOTE] PIP-196 Segmented transaction buffer snapshot
+1(non-binding) Hang Chen 于2022年9月13日周二 11:28写道： > +1 (binding) > > Thanks, > Hang > > guo jiwei 于2022年9月9日周五 18:17写道： > > > > +1(binding) > > > > Great work ! > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Fri, Sep 9, 2022 at 10:10 AM PengHui Li wrote: > > > > > +1(binding) > > > > > > I have done the review on gdoc > > > And please also update the github issue(PIP). > > > > > > Thanks, > > > Penghui > > > > > > On Fri, Sep 9, 2022 at 9:31 AM 丛搏 wrote: > > > > > > > Hi, Xiangying > > > > +1(non-binding) > > > > > > > > This PIP overall LGTM! It solves the problem of snapshots not being > > > > able to scale. > > > > We can make some optimizations later : > > > > 1. Merge transaction snapshot segments to reduce the number of > > > > segments and the index's size. > > > > 2. Add snapshot segments to memory as TB requires reducing memory > > > overhead. > > > > > > > > Thanks! > > > > Bo > > > > > > > > Xiangying Meng 于2022年9月8日周四 22:17写道： > > > > > > > > > > Hi, community > > > > > This proposal has some updates. The latest version of the proposal > can > > > be > > > > > found here > > > > > < > > > > > > > > https://docs.google.com/document/d/1hBk2nGcj0Os-ULi2q404gCoxIsPleGY8p5H1hqWD5kI/edit# > > > > > > > > > > . > > > > > Feel free to comment on this doc. > > > > > Sincerely, > > > > > Xiangying > > > > > > > > > > On Wed, Sep 7, 2022 at 4:55 PM Xiangying Meng < > xiangy...@apache.org> > > > > wrote: > > > > > > > > > > > Hi, community > > > > > > I,d like to start a vote for the PIP-196 > > > > > > <https://github.com/apache/pulsar/issues/16913> Segmented > > > transaction > > > > > > buffer snapshot. > > > > > > And the discussion can be found here > > > > > > < > https://lists.apache.org/thread/bqoy3oz8flvxy7xpmnw81cr4c9sz5vy0>. > > > > > > > > > > > > Sincerely, > > > > > > Xiangying > > > > > > > > > > > > > > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] PIP 194 : Pulsar client: seek command add epoch
de to reset the position of the > *subscription*. When that happens, the broker decides, again, to reset all > existing TCP connections to all consumers upon receiving the seek command, > and you would expect any messages sent afterward to be from the new > position, which again doesn't happen. > > Another really important piece of information we need to bring to the > context of the reader here is the notion of an epoch. First, the epoch in > Pulsar PIPs was introduced in PIP-84. The idea is that every time the > client starts a "session" of requesting and receiving messages in response, > the client will send a Session Sequence Number, and the server responds to > those message requests with the same session sequence number. Since Pulsar > doesn't follow a request-response model but has a bi-directional protocol, > the client can send a command to fetch messages using a new session > sequence number, while the server can still send messages using the old > session number. Using the Session Sequence Number the client can't tell the > difference between the messages being pushed from the server to it. That > Session Sequence Number has the one referred to as Epoch in PIP-84 and also > in this PIP. > The idea was somehow to demarcate the responses coming from the server > based on the commands the client sends as they are *independent* (async). > > # What are the issues with this PIP? > 1. The PIP decides to solve the problem listed above *only* for exclusive > and failover subscriptions where you have only a single consumer. The > problem still remains at large with Shared or Key Shared subscriptions. > 2. The cost of solving a small portion of the problem is high: > Added Complexity - Adding another field to the protocol, and another > thing to check. I believe we should aim to reduce the cognitive load of the > developers of Pulsar. > 3. There are no rejected solutions - We always need to examine all > available options and list why we decided against them. > 4. Lack of background knowledge (context) - it's super hard IMO to grasp > the idea without so much context missing: The client-server protocol > pertaining to this PIP, including its async nature, what is an epoch and > why it was introduced, what are flow permits. I'm not saying explain all > pulsar in this doc, but just include a brief explanation of that > terminology. > > # What We Suggest > > Rethink the solution. > 1. The consumer (one of many) will send a seek command to the broker, and > at the same time clear its internal queue and wait for a response from the > broker. > 2. The broker upon receiving the seek command, will > a. Stop sending dispatching messages to consumers. > b. Notify all consumers via a command (new) that the subscription > position was asked to be reset. Consumers receiving this command will clear > their internal queue. The broker will no longer close the TCP connection > (with its adverse effects on other consumers and produces "riding" on that > connection) > c. Reset the cursor to the newly requested position. > d. Continue dispatching messages from newly requested positions to > consumers. > > The disadvantages here are that we need to alter the client to get to know > a new command and act accordingly, yet I think that is accidental > complexity stemming from the client-server architecture of bi-directional > and not request response. > > Thanks, > > Asaf > > On Mon, Aug 1, 2022 at 6:43 AM Qiang Huang > wrote: > > > Sure. You can refer to pip-84: > > > > > https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch > > . > > > > Zike Yang 于2022年7月29日周五 10:22写道： > > > > > Hi, Qiang > > > > > > > It is necessary to check the current cursor status when handling > > > flowPermits > > > > request from the server side. If the server is handling seek request, > > it > > > > should ignore flowPermits request because the request is illegal. > > > > > > Thanks for your explanation. I think it's better to add this > > > explanation to the PIP. > > > > > > > The reconnected consumer can regard as a new consumer with new epoch. > > > > > > The consumer will reconnect to the broker during the seek operation. > > > And this will change the existing behavior. It doesn't seem to make > > > sense. Please correct me if I have misunderstood. > > > > > > Thanks, > > > Zike Yang > > > > > > On Wed, Jul 27, 2022 at 8:06 PM Qiang Huang > > > > wrote: > > > > > > > > Thanks Zike. > > &
Re: [VOTE] Pulsar Release 2.7.5 Candidate 3
+1(non-binding) - Checked signatures/checksums - Checked the license headers - Ran the standalone - Validate Connectors - Validate Pub/Sub and Java Functions - Validate Stateful Functions Haiting Jiang 于2022年8月27日周六 06:40写道： > This is the third release candidate for Apache Pulsar, version 2.7.5. > > It fixes the following issues: > https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-3 > > *** Please download, test and vote on this release. This vote will stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/ > > SHA-512 checksums: > > > 6aa74cb742c411edd40de2199eb70d774d0ce8cb7dbfc2ce33fce7e87949aafa73f93a1e154261b58a1df4ad10160bf43fc5a93b1aa9a0466fcf2a3ba1a6e385 > apache-pulsar-2.7.5-bin.tar.gz > > facb1698e394b5428a0b9b6d71b891013f3c5b473cf72dd68a75ca729800710f2c6c476b1c47f7804c1b1c0941f79a6a9f334c7c703f66f6bac8f399386c5ad6 > apache-pulsar-2.7.5-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1172/ > > The tag to be voted upon: > v2.7.5-candidate-3 (8eae5b8d572861e49c40d456b1f3cbc5d414afe1) > https://github.com/apache/pulsar/releases/tag/v2.7.5-candidate-3 > > Docker images: > > > https://hub.docker.com/layers/jason918/pulsar/2.7.5/images/sha256-ec77603bd943f1a56065d155428d6edccb2bf2ec57fafab8112c4b73c7bfcb6e > > https://hub.docker.com/layers/pulsar-all/jason918/pulsar-all/latest/images/sha256-0792a0ec75ecbc16c9a42af127a68b3dea2331be3f2f0599f5fb1f4cd78def25 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download the source package, and follow the README to build > and run the Pulsar standalone service. > > Thanks, > Haiting > -- BR, Qiang Huang
Re: [DISCUSS] Deprecate KeyValue schema factory methods with Class parameters
Nice catch. LGTM. Yunze Xu 于2022年8月24日周三 16:29写道： > Hi folks, > > Recently I'm looking into the KeyValue schema and found **FOUR** > static methods in `Schema` to create a `KeyValueSchema` object: > > 1. KeyValue(Class key, Class value); > 2. KeyValue(Class key, Class value, SchemaType type); > 3. KeyValue(Schema key, Schema value); > 4. KeyValue(Schema key, Schema value, KeyValueEncodingType > keyValueEncodingType); > > IMO, having too many overloads is not user-friendly. I turned to the > official website and found overload 4 is used. The overload 3 is just > a simple version of overload 4 whose encoding type is `INLINE`, but > it's not clear. I opened a PR > https://github.com/apache/pulsar/pull/17256 to make it more clear. > > However, for overload 1 and 2, I can only find two references in unit > tests `testAllowNullAvroSchemaCreate` and > `testAllowNullJsonSchemaCreate`. From the very simple JavaDocs, it > looks like they are only available for JSON and AVRO schemas. > > Then I found the original PR to introduce these overloads: > https://github.com/apache/pulsar/pull/2885 > > Code has been changed a lot since that. It looks other codes are not > much related to these two overloads now. IMO, we should not encourage > users to use these two overloads, so I suggest marking them as > deprecated and might remove them in future releases. > > > Thanks, > Yunze > > > > > -- BR, Qiang Huang
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (non binding) Yunze Xu 于2022年8月24日周三 15:44写道： > +1 (non binding) > > Thanks, > Yunze > > > > > > 2022年8月24日 15:38，Nicolò Boschi 写道： > > > > +1 (non binding) > > Nicolò Boschi > > > > > > Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < > > eolive...@gmail.com> ha scritto: > > > >> Hello, > >> this is the official thread VOTE for PIP-201 Extensions mechanism for > >> Pulsar Admin CLI tools > >> > >> This is the PIP link https://github.com/apache/pulsar/issues/17155 > >> This is the discussion: > >> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo > >> > >> I am still working on the PR, that is the subject of the VOTE. > >> > >> Best regards > >> Enrico Olivelli > >> > > -- BR, Qiang Huang
Re: [VOTE] [PIP-169] Support split bundle by flow or qps
+1(non-binding) Enrico Olivelli 于2022年8月24日周三 15:04写道： > +1 (binding) > > > Enrico > > Il giorno mer 24 ago 2022 alle ore 07:13 Aloys Zhang > ha scritto: > > > > +1 > > > > Haiting Jiang 于2022年8月24日周三 12:35写道： > > > > > +1 (non) > > > > > > Thanks, > > > Haiting > > > > > > On Wed, Aug 24, 2022 at 10:19 AM guo jiwei > wrote: > > > > > > > +1 (binding) > > > > > > > > > > > > Regards > > > > Jiwei Guo (Tboy) > > > > > > > > > > > > On Wed, Aug 24, 2022 at 1:01 AM Heesung Sohn > > > > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > Heesung > > > > > > > > > > On Tue, Aug 23, 2022 at 10:00 AM Anon Hxy > wrote: > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > Thanks, > > > > > > Xiaoyu Hou > > > > > > > > > > > > lordcheng10 <1572139...@qq.com.invalid> 于2022年8月23日周二 15:10写道： > > > > > > > > > > > > > Hi Pulsar Community, I would like to start a VOTE on "Support > split > > > > > bundle > > > > > > > by flow or qps."(PIP-169) > > > > > > > Here is the design detail: > > > > > https://github.com/apache/pulsar/issues/16782 > > > > > > > > > > > > > > > > > > > > > and the discussion thread: > > > > > > > > https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > lordcheng10 > > > > > > > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Minor breaking change to 2 method signatures in RangeCache class
Great work! It can tune the range cache better. Michael Marshall 于2022年8月24日周三 11:11写道： > Hi All, > > I'd like to add some more metrics to the broker entry cache. In order > to do so, I needed to update RangeCache#clear and > RangeCache#evictLEntriesBeforeTimestamp so they return Pair Long> instead of long. This is technically a breaking change, but > since it is an internal broker class, it would only break broker > extensions, if they used the return value for these methods. > > Please let me know if you think the changes will be a problem. > > Here is the PR: https://github.com/apache/pulsar/pull/17248. > > Thanks, > Michael > -- BR, Qiang Huang
Re: [DISCUSS] Apache Pulsar 2.10.2 release
+1 PengHui Li 于2022年8月23日周二 23:42写道： > +1 > > And, I have added the release/2.10.2 label, Lan Liang. > > Thanks, > Penghui > > On Tue, Aug 23, 2022 at 11:02 PM Lan Liang > wrote: > > > Hi,Haiting: > > > >Thanks for your work of release. Could you help to release PR-16309 > > [0] to 2.10.2, It resolved pulsar can not working with etcd cluster. > > > >Some users are wondering about this part [1]. > > > > [0] https://github.com/apache/pulsar/pull/16309 > > [1] https://github.com/apache/pulsar/discussions/17240 > > > > > > > > > > Best Regards, > > Lan Liang > > Replied Message > > | From | Haiting Jiang | > > | Date | 8/23/2022 22:10 | > > | To | | > > | Subject | [DISCUSS] Apache Pulsar 2.10.2 release | > > Hello, Pulsar community: > > > > I'd like to propose to release Apache Pulsar 2.10.2 > > > > Currently, we have 189 commits [0] labeled with `release/2.10.2`, > > And 39 of them are not cherry-picked yet [1]. > > > > And there are 39 open PRs [2], I will follow them to make sure that > > the important fixes could be contained in 2.10.2. > > > > If you have any important fixes or any questions, > > please reply to this email, we will evaluate whether to > > include it in 2.10.2 > > > > [0] > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+ > > [1] > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+-label%3Acherry-picked%2Fbranch-2.10+ > > [2] > > > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.2+ > > > > Best Regards > > Haiting Jiang > > > -- BR, Qiang Huang
Re: [DISCUSS] Make `Pulsar CI / CI - Unit - Brokers - Flaky` test group to be required
Failure flaky tests reduce the passion of contributors. I suggest making stable tests as required. I have 3 points. 1. Find the existing failure flaky tests and fix them. I believe that there is already a project to record them. 2. Move the failure flaky tests out of the flaky test group. 3. Add them into a new group and fix. And make it required if they are stable. PengHui Li 于2022年8月22日周一 17:27写道： > I agree with moving the tests out of the flaky test group. > I just checking some new PRs > > https://github.com/apache/pulsar/pull/17195 > https://github.com/apache/pulsar/pull/17201 > https://github.com/apache/pulsar/pull/17193 > https://github.com/apache/pulsar/pull/17204 > > The `Pulsar CI / CI - Unit - Brokers - Flaky` looks more stable than other > groups such as > > `Pulsar CI / CI - System - Pulsar Connectors - Thread` > `Pulsar CI / CI - Unit - Brokers - Broker Group 2` > `Pulsar CI / CI - Unit - Brokers - Broker Group 1` > > It looks like we can change the test group for now and make it required. > To a new test group, or move them to `Pulsar CI / CI - Unit - Brokers - * ` > > Thanks, > Penghui > > On Mon, Aug 22, 2022 at 4:33 PM tison wrote: > > > While agree to require more tests to pass, here are my two coins: > > > > 1. This group is named "flaky tests" so I regard it as flaky tests > > literally. NOT require these tests to pass could be by design. Besides, > > IIRC some developers keep investigating tests in the flaky test group, > try > > to make them stable, and move out of the flaky test group. This seems the > > desired approach to resolve flaky tests. > > > > 2. Instead of barely "require broker-flaky" test group, do you have a > list > > of tests that fail frequently? Otherwise, we just go back to the > situation > > where we want to exclude it from the required status - it's quite > unstable. > > > > Best, > > tison. > > > > > > mattison chao 于2022年8月22日周一 15:56写道： > > > > > Hi all > > > > > > Recently, some tests in the `broker-flaky` test group always failed, > but > > > since it doesn't block CI, no one cared for a long time. > > > > > > This behaviour causes some test scenarios to go unchecked and risk > > > introducing some regressions, and I think we need to make this test > group > > > set required. > > > > > > e.g. https://github.com/apache/pulsar/pull/17163 > > > > > > WDYT? > > > > > > Best, > > > Mattison > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Extended auth operation
Nice catch! +1 Zixuan Liu 于2022年8月22日周一 16:49写道： > Hi all, > > I noticed we added the TenantOperation, NamespaceOperation, and > PolicyOperation in https://github.com/apache/pulsar/pull/6428, which > provide the "real" authz abilities for each resource. This PR only adds > check permissions method in authorization provider, no grant, revoke, and > get action, so I want to add this feature. > > Best, > Zixuan > -- BR, Qiang Huang
Re: [DISCUSS] Use Github project to track the multi clients development
It makes sense to me. I'm thinking, if an issue is encountered in a Java client, it is possible to be in other clients. If so, the Java client board is also needed. mattison chao 于2022年8月22日周一 16:00写道： > +1 > > Best, > Mattison > > On Mon, 22 Aug 2022 at 15:52, Baodi Shi > wrote: > > > Nice idea! > > > > Thanks, > > Baodi Shi > > > > > On Aug 22, 2022, at 12:4250, Zike Yang wrote: > > > > > > Hi, all > > > > > > Currently, many features or bug fixes of some clients like the C++ > > > client and Python client have not caught up with the Java client. For > > > better tracking of these features or bug fixes, I have created a > > > Github project called `Apache Pulsar / Multi Clients`[0] under the > > > pulsar main repo. It can be removed if we don't like it this way. > > > > > > There are six columns in this Github project: > > > * Backlog: Low priority issues or issues(or cards) that have not been > > > initially investigated. > > > * Todo: Newly added issues and issues suitable for contributors to pick > > first. > > > * In progress: Work in progress issues or PRs. > > > * Review in progress: The PRs or issues that are waiting for review. > > > * Reviewer approved: The PRs or issues that are waiting to merge. > > > * Done: Merged PRs and Solved issues. > > > > > > This not only helps to track the status of other clients' features and > > > bug fixes but also helps contributors who want to participate in other > > > language client feature development to pick the task. > > > > > > This Project is currently only for clients maintained by the main > > > pulsar repository, such as C++ and Python clients. > > > > > > I have created a PR [1] to add C++ and Python client issues and PRs to > > > the Multi Clients project. > > > > > > Please let me know what you think. Thanks! > > > > > > [0] https://github.com/apache/pulsar/projects/12 > > > [1] https://github.com/apache/pulsar/pull/17201 > > > > > > Best, > > > Zike Yang > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Enable message deduplication for replicator by default
+1 Zike Yang 于2022年8月22日周一 15:32写道： > +1 > > Thanks, > Zike Yang > > On Mon, Aug 22, 2022 at 3:16 PM mattison chao > wrote: > > > > +1 > > > > Best, > > Mattison > > > > On Fri, 19 Aug 2022 at 01:40, Enrico Olivelli > wrote: > > > > > I agree > > > > > > Enrico > > > > > > Il Gio 18 Ago 2022, 18:23 PengHui Li ha scritto: > > > > > > > Hi all, > > > > > > > > When I tried to fix a problem related to replicator > > > > https://github.com/apache/pulsar/pull/17154 > > > > It surprised me that the message deduplication will not work by > default > > > > with the replicator. > > > > I always thought it was enabled for replicators by default. Details > to > > > see > > > > [0]. > > > > > > > > I think we should enable the deduplication for the replicator. > Otherwise, > > > > we will see duplicated > > > > messages on the remote cluster. And the producer of the replicator > always > > > > has a fixed producer > > > > name, this will make the message deduplication work properly. > > > > > > > > The test introduced in https://github.com/apache/pulsar/pull/17154 > will > > > > check the message > > > > replication ordering. Without the message deduplication enabled, the > test > > > > is flaky with received > > > > duplicated messages. After enabling, everything is fine. > > > > > > > > Best, > > > > Penghui > > > > > > > > [0] > https://github.com/apache/pulsar/pull/17154#discussion_r948736894 > > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Enable non-mandatory updating PR branches
It works now. It helps. Zixuan Liu 于2022年8月22日周一 10:38写道： > +1 > > Best, > Zixuan > > On 2022/08/18 13:01:58 tison wrote: > > Hello, > > > > The short version > > = > > > > Vote if you agree on enabling the non-mandatory updating PR branches > > button, i.e., the "Always suggest updating pull request branches" GitHub > > settings. > > > > The full version > > > > > > Pulsar is under rapid development and numerous fixes are pushed to master > > every time. Since we are still suffering from quite a few flaky tests, > > merge master and retest is a hotspot to verify the patch once more. > > > > However, we should pull the nightly master locally, check out the PR > > branch, perform the merge and push to remote. It's a bit awkward > especially > > when a developer works on multiple branches. > > > > GitHub provides a button "Always suggest updating pull request branches" > > with the description "Whenever there are new changes available in the > base > > branch, present an “update branch” option in the pull request."[1] > > > > It can simplify the workflow with one button click. > > > > To clarify, this is different from the branch protection rule "Require > > branches to be up to date before merging" - it's non-mandatory and just > > provides the "update branch" button. It means we don't force every PR to > > catch up with the latest master before merged, which can cause > exextremely > > high unnecessary traffic. Since we already allow PR authors to retrigger > > tests with the pulsorbot (or even contributors can push an empty commit), > > providing such a button does no harm. > > > > I post this thread here to collect feedback, especially from the PMC > > members. Previously I asked the INFRA team to turn on this option for > > Apache Kvrocks (Incubating)[2] and I believe the INFRA team would be > happy > > to see an explicit community consensus. > > > > Best, > > tison. > > > > [1] > > > https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/ > > [2] https://issues.apache.org/jira/browse/INFRA-23432 > > -- BR, Qiang Huang
Re: Enable "Update branch" Github button in pull requests
+1. It can help a lot. Zixuan Liu 于2022年8月22日周一 10:38写道： > +1 > > Best, > Zixuan > > On 2022/07/12 16:06:43 Nicolò Boschi wrote: > > Hi all, > > > > I'd like to propose to enable the Github button "update branch" in the > pull > > requests. > > > > The main reason is that it helps when you want to rebase your pull to the > > current master. > > For instance today a fix for a failing job in the CI has been committed > and > > you were forced to close/reopen the pull or manual rebase your branch. > With > > this button is much easier and it will retrigger the CI as well. > > > > This is the guide to enable it: > > > https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/keeping-your-pull-request-in-sync-with-the-base-branch > > > > This is an example where I activated the button in my own Pulsar fork: > > https://github.com/nicoloboschi/pulsar/pull/4 > > > > Thanks, > > Nicolò Boschi > > -- BR, Qiang Huang
Re: [ANNOUNCE] Jiwei Guo as a new PMC member in Pulsar
Congratulations Jiwei!!! Jiuming Tao 于2022年8月19日周五 11:32写道： > Congratulations! > > Yu 于2022年8月19日周五 11:02写道： > > > Congrats Jiwei! > > > > On Fri, Aug 19, 2022 at 10:45 AM Zixuan Liu wrote: > > > > > Congrats! > > > > > > Zixuan > > > > > > Jun M 于2022年8月19日周五 10:16写道： > > > > > > > Congrats to Jiwei! > > > > > > > > > > > > Cheers, > > > > Jun > > > > > > > > > > -- BR, Qiang Huang
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
Great work. I prefer "Choice A". > Cherry pick changes [4] > Choice A: [fix][broker][branch-2.9] xxx Yunze Xu 于2022年8月17日周三 18:32写道： > LGTM. > > Thanks, > Yunze > > > > > > 2022年8月17日 11:15，Yu 写道： > > > > Hi team, > > > > For PIP 198: Standardize PR Naming Convention using GitHub Actions [1] > > > > How to define [type] and [scope]? Do these abbreviations LGTY? > > > > *[Guide] Pulsar Pull Request Naming Convention* [2] contains everything > > about the definition. Feel free to check and comment! > > > > ~~ > > > > TL;DR > > > > PR title format: [type][scope] Summary [3] > > > > ~~ > > > > [type] > > > > 1. Definition: what actions do you take? > > > > 2. It must be one of the following: > > - feat (abbr for "feature") > > - improve > > - fix > > - cleanup > > - refactor > > - revert > > > > ~~ > > > > [scope] > > > > 1. Definition: where do you make changes? > > > > 2. It must be one of the following: > > - admin (changes to pulsar-admin, REST API, Java admin API) > > - broker > > - io > > - deploy > > - dep (abbr for dependency) > > - fcn (abbr for function) > > - monitor > > - pkg (abbr for package) > > - proxy > > - schema > > - sec (abbr for security) > > - sql > > - ts (abbr for tiered storage) > > - tool > > - txn (abbr for transaction) > > > > - java (changes to Java client) > > - cpp (changes to C++ client) > > - py (changes to Python client) > > - ws (changes to WebSocket) > > - rest (changes to REST) > > > > - test > > - ci > > - workflow > > - build > > - misc (abbr for miscellaneous) > > > > - doc > > - blog > > - site (abbr for website) > > > > ~~ > > > > Besides, many developers have different opinions on the following > aspects. > > What's your writing preference? > > > > - Submit breaking changes > > [feat][broker]! Support xx > > > > - Submit PIP changes > > [feat][broker] PIP-198: Support xx > > > > - Cherry pick changes [4] > > Choice A: [fix][broker][branch-2.9] xxx > > Choice B: [fix][broker] xxx. And add "cherry pick xxx to branch-2.9" in > the > > PR description. > > > > ~~ > > > > Feel free to comment and make your voice heard. Go vote! Thank you! > > > > [1] > > > https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit > > [2] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn > > [3] > > > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno > > [4] > > > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=kix.849jztd92uk7 > > > > Yu > > -- BR, Qiang Huang
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Looks good. I have two points: 1. It is necessary to supplement the upgrade and downgrade documentation in Pulsar. 2. There are 3 issues mentioned in the PIP, should we split it into 3 small issues? Enrico Olivelli 于2022年8月17日周三 17:30写道： > I generally agree with the PIP > > Can you please explain the interactions with the Pulsar Helm chart ? > also we have to draw a migration path, because users that will upgrade > Pulsar will have to move the configuration files in another location > > Enrico > > Il giorno mer 17 ago 2022 alle ore 11:15 tison > ha scritto: > > > > Hello, > > > > This is a PIP to package the Pulsar Trino distro and config in a > dedicated > > folder. > > > > Link: https://github.com/apache/pulsar/issues/17137 > > Prototype: https://github.com/apache/pulsar/pull/17062 > > > > Below you can find the proposal (I will amend the GH issue while we > discuss > > it). > > > > Best, > > tison. > > > > Motivation > > > > > > After https://github.com/apache/pulsar/pull/16683 merged, we upgrade > > PrestoSQL dependency in Pulsar SQL to the first several Trino version. To > > handle the name change cases and gradually refactor Pulsar SQL as a > > self-contained module so that we can move it into a standalone > repository, > > I find that there're three major issues to resolve. > > > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with other > > Pulsar configs. > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL. > > 3. Integration tests of Pulsar SQL are tightly coupled with the main repo > > (test infra). > > > > This proposal is aimed at resolving the first issue to package Pulsar > Trino > > distro and config in a dedicated folder; that is, to make it > self-contained. > > > > Goal > > > > > > I have already prepared a draft to perform the changes as > > https://github.com/apache/pulsar/pull/17062. Generally, we move the > config > > files under `PRESTO_HOME` and correspondingly update scripts. > > > > In this way, all Trino distro artifacts are under the same home path, so > > that we can later move it out as a whole. > > > > This change should not affect those who use Pulsar with the entry point > > script, but it changes the layout of the release artifact, so I'd prefer > to > > perform a PIP process. > > > > Implementation > > > > > > It's straightforward to inline in the "Goal" section. > > > > However, the name of the folder (`presto` or `trino`) and the level of > the > > folder (`lib/presto/` or `trino/`) is open to discussion. I think both > are > > fine and will try `trino/` first. > > > > To minimize unnecessary changes, I tend to keep the modules name > > `pulsar-presto-xxx` as is. > > > > Alternatives > > = > > > > I don't make a completed proposal to resolve all three issues listed > above. > > Because I'm still unfamiliar with the latter two topics yet and I'd > prefer > > to implement these improvements one by one since they're naturally > > independent. If I try to make a completed proposal at once, it's highly > > possible I give up halfway. > > > > Anything else? > > === > > > > Previous discussion: > > > > [DISCUSS] Move Pulsar SQL to a separated repository? > > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8 > -- BR, Qiang Huang
Re: Spring for Apache Pulsar
Great. It makes the pulsar ecosystem wider and wider tison 于2022年8月17日周三 00:42写道： > Cool! I can't wait to give it a try. > > Soby Chacko 于2022年8月16日 周二22:41写道： > > > Apologies for the duplicate email. I missed adding a subject in my last > > email. > > > > Thank you. > > > > -- Forwarded message - > > From: Soby Chacko > > Date: Tue, Aug 16, 2022 at 10:39 AM > > Subject: > > To: > > > > > > Hi Pulsar community, > > > > Cross posting this from Slack. > > > > We are happy to announce a new incubating Spring project for Apache > Pulsar. > > > > https://github.com/spring-projects-experimental/spring-pulsar > > > > More info can be found on this blog: > > > > > https://spring.io/blog/2022/08/16/introducing-experimental-spring-support-for-apache-pulsar > > > > Looking forward to any feedback, feature request and PR contributions!! > > > > Best Regards, > > Soby Chacko > > > -- > Best, > tison. > -- BR, Qiang Huang
Re: [DISCUSS] Add an auth data const for refresh the original auth data
It makes sense to me. BTW, the image is broken. Zixuan Liu 于2022年8月17日周三 11:10写道： > Note that there are two clients, the user client, and the proxy client. > When the original authenticate data expires, the user client cannot send a > request to the proxy to find the broker URL. We haven't tests to cover this. > > A simple diagram represents workflow: > [image: image.png] > Both connections pass the proxy client and the user client authentication > data. > > Thanks, > Zixuan > > Zixuan Liu 于2022年8月16日周二 23:02写道： > >> Hi all, >> >> Refreshing the authentication data comes from the client is important. We >> have two types of authentication data, directly authentication data, and >> original authentication data: >> >> 1. Directly authentication data >> The client/proxy brings the authentication data directly connected to the >> broker, which is directly authentication data. >> >> When the directly authentication data is expired, the broker sends the >> `newAuthChallenge` command with `AuthData.REFRESH_AUTH_DATA` data to the >> client to refresh the authentication data. >> >> 2. Original authentication data >> We add a proxy between the client and the broker, both the proxy and the >> client bring the authentication data to request the broker, the >> authentication data from the proxy is directly authentication data, and the >> authentication data from the client is original authentication data. >> >> The broker can refresh the directly authentication data, but when we are >> using the proxy, the broker could not refresh the original >> authentication data, because we haven't any action to request to refresh >> the original authentication data, so we need to add an auth data const to >> request to refresh the original authentication data, so like >> `AuthData.REFRESH_AUTH_DATA`. >> >> Once most people agree with this, I'll make a PIP. >> >> References: >> >> - https://github.com/apache/pulsar/pull/13339 >> - https://github.com/apache/pulsar/issues/10816 >> >> Thanks, >> Zixuan >> >> -- BR, Qiang Huang
Re: [DISCUSS] Support `brokerMaxConnectionsPerIp` on Pulsar proxy
Good idea. The implementation should provide an appropriate exception to prevent the client from reconnecting continuously if it reaches the limitation. Do other modules also need this? Michael Marshall 于2022年8月16日周二 13:52写道： > Good idea, it makes sense to me to add this to the proxy. > > > BTW, are there any side effects from the implementation? > > Reading through the code from > https://github.com/apache/pulsar/pull/10754, it looks like the cost is > minimal. The broker maintains counters for each IP address, and they > are incremented/decremented when a ServerCnx goes active/inactive. > > Thanks, > Michael > > On Mon, Aug 15, 2022 at 11:27 PM Haiting Jiang > wrote: > > > > Sounds good. It's good for server stability. > > > > BTW, are there any side effects from the implementation? > > > > On Tue, Aug 16, 2022 at 10:33 AM mattison chao > > wrote: > > > > > Hi, All > > > > > > Pulsar has the `brokerMaxConnectionsPerIp` configuration at the > > > broker, we can use it to limit the maximum connections per IP. > > > > > > The original motivation and PR here: > > > https://github.com/apache/pulsar/pull/10754 > > > > > > IMO, we can also apply it to pulsar-proxy, because when a large > > > number of proxy accesses under the same IP (maybe due to some wrong > > > operations) will cause the proxy to accept too much wrong traffic and > > > cause service unstable. > > > > > > WDYT? > > > > > > Best, > > > Mattison > > > > -- BR, Qiang Huang
Re: [VOTE] PIP-193 : Sink preprocessing Function
+1 (non binding) Nicolò Boschi 于2022年8月16日周二 17:28写道： > +1 (non binding) > > Thanks, > Nicolò Boschi > > > Il giorno mer 10 ago 2022 alle ore 01:34 Neng Lu ha > scritto: > > > Hi, > > > > Not sure if this is too late or not, I replied in the discussion thread > > about some thinking. > > Whether we tweak the sink connector or we allow a flexible and general > > function creation. > > > > On Mon, Aug 1, 2022 at 5:03 PM mattison chao > > wrote: > > > > > +1 (non-binding) > > > > > > Best, > > > Mattison > > > > > > On Tue, 2 Aug 2022 at 01:56, Dave Fisher wrote: > > > > > > > > +1 (binding) > > > > > > > > On 2022/07/28 10:39:35 Christophe Bornet wrote: > > > > > Hi, Pulsar community, > > > > > > > > > > I'd like to start a vote on PIP-193 : Sink preprocessing Function > > > > > > > > > > You can find the proposal at > > > https://github.com/apache/pulsar/issues/16739 and > > > > > the discussion thread at > > > > > https://lists.apache.org/thread/qn59jwn47w9ngxpkvq3kswbl1y882jth. > > > > > > > > > > The vote will stay open for at least 48 hours. > > > > > > > > > > Best regards. > > > > > > > > > > Christophe Bornet > > > > > > > > > > > > > > -- > > Best Regards, > > Neng > > > -- BR, Qiang Huang
Re: [DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface
> might be asynchronous, and having support for a state machine for > creation > > and deletion could be helpful. This is to bring up the point that it's > not > > optimal to model a topic deletion as an atomic operation. The state > change > > should be atomic, but the deletion from the metadata storage should not > > happen until all asynchronous operations have been completed. The > metadata > > admin interface caller should be able to proceed after it is marked > > deleted, but the system should keep on managing the deletions in the > > background. Similarly, the creation of topics could have more states to > > deal with efficient creation of a large number of topics. > > > > This was a long email covering a subject that we haven't dealt with > before > > in the Pulsar community. Usually, we have discussions about solutions > that > > are very targeted. It isn't common to transparently discuss existing > > design challenges or problems and find ways to solve them together. > Sharing > > observations about problems would be valuable. High-level problems don't > > get reported in the GitHub issue tracker since they aren't individual > bugs. > > We should find ways to address also this type of challenges in the > > community. > > > > I hope we can change this and also take the opportunity to meet at Pulsar > > Community meetings and have more of these in-depth discussions that will > > help us improve Pulsar for the benefit of us all in the Apache Pulsar > > community. > > > > Since PIP-157 [3] is proceeding, I see that as an opportunity to start > > taking the design of Pulsar Metadata handling in the direction where we > > could address the challenges that there are currently in Pulsar with > > metadata handling and load balancing. We must decide together what that > > direction is. I hope this email opens some new aspects to the basis of > > these decisions. I'm hoping that you, the reader of the email, > > participate to share your views and also help develop this direction. > > > > PIP-157 [3] assumes "Pulsar is able to manage millions of topics but the > > number of topics within a single namespace is limited by metadata > > storage.". Does this assumption hold? > > > > For example, "3) Scalability issue: all metadata changes are broadcasted > > to all brokers" will become a challenge in a large system with a high > > number of brokers. Together with the other Metadata consistency > challenges > > ( 1 and 2 above), I have a doubt that after PIP-157 is implemented, the > > bottlenecks will move to these areas. In that sense, it might be a > band-aid > > that won't address the root cause of the Pulsar Metadata handling > > scalability challenges. > > > > Let's discuss and address the challenges together! > > > > Regards, > > > > -Lari > > > > [1] - analysis about Metadata consistency from user's point of view - > > https://github.com/apache/pulsar/issues/12555#issuecomment-955748744 > > [2] - MetadataStore interface: > > > https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/api/MetadataStore.java > > [3] - PIP-157: Bucketing topic metadata to allow more topics per > namespace > > - https://github.com/apache/pulsar/issues/15254 > > [4] - PIP-45: Pluggable metadata interface - > > > https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface > > > > [5] - StreamNative's blog "Moving Toward a ZooKeeper-Less Apache Pulsar" > - > > > https://streamnative.io/blog/release/2022-01-25-moving-toward-a-zookeeperless-apache-pulsar/ > > > -- BR, Qiang Huang
Re: [VOTE] PIP-186: Introduce two phase deletion protocol based on system topic
+1(non-binding) horizonzy 于2022年8月16日周二 16:21写道： > Dear Community, > > I'd like to start a VOTE on "PIP-186: Introduce two phase deletion protocol > based on system topic" > > The proposal can be read at [0] and the discussion thread is available at > [1] > > Voting will stay open for at least 48h. > > [0] https://github.com/apache/pulsar/issues/16569 > [1] https://lists.apache.org/thread/ftr7qhxhglso3bnsv8lvlgoxmx419ymg > > Thanks. > -- BR, Qiang Huang
Re: [VOTE] PIP-191: Support batched message using entry filter
+1(non-binding) Anon Hxy 于2022年8月16日周二 13:11写道： > Dear Community, > > I'd like to start a VOTE on "PIP-191: Support batched message using entry > filter." > > The proposal can be read at [0] and the discussion thread is available at > [1] > > Voting will stay open for at least 48h. > > [0] https://github.com/apache/pulsar/issues/16680 > [1] https://lists.apache.org/thread/cdw5c2lpj5nwzl2zqyv8mphsqqv9vozj > > Thanks, > Xiaoyu > -- BR, Qiang Huang
Re: [VOTE] Pulsar Release 2.8.4 Candidate 1
Got it. Thx. Yunze Xu 于2022年8月14日周日 23:22写道： > You can see > https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f > for more details about the LTS release. > > Thanks, > Yunze > > > > > > 2022年8月14日 11:00，Qiang Huang 写道： > > > > +1 (non-binding) > > Is 2.8.4 a long term support release? > > > > Yunze Xu 于2022年8月12日周五 16:20写道： > > > >> This is the first release candidate for Apache Pulsar, version 2.8.4. > >> > >> It fixes the following issues: > >> > >> > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.8.4 > >> > >> *** Please download, test and vote on this release. This vote will stay > >> open > >> for at least 72 hours *** > >> > >> Note that we are voting upon the source (tag), binaries are provided for > >> convenience. > >> > >> Source and binary files: > >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.4-candidate-1/ > >> > >> SHA-512 checksums: > >> > >> > c3d26704f2cfb3365c29d4110612ca7351084f8bee3c306d5e906b3f9b22c7557cc5baf12f74f8c222baccae3310691419eda5b47fdf9cd6c5281b70134ab5eb > >> apache-pulsar-2.8.4-bin.tar.gz > >> > 28160ee94dccfb74dfb56e0e5d0e08870c6612659507333a52b5660ecbf060a75d1eed667cffd8596f9468de95055b78916b932db0e0d4c2745868d55429ee98 > >> apache-pulsar-2.8.4-src.tar.gz > >> > >> Maven staging repo: > >> > https://repository.apache.org/content/repositories/orgapachepulsar-1170/ > >> > >> The tag to be voted upon: > >> v2.8.4-candidate-1 (02ee5616866d4eda8dd94f85d9d9b71c459f248d) > >> https://github.com/apache/pulsar/releases/tag/v2.8.4-candidate-1 > >> > >> Pulsar's KEYS file containing PGP keys we use to sign the release: > >> https://dist.apache.org/repos/dist/dev/pulsar/KEYS > >> > >> Docker images: > >> > >> > >> > https://hub.docker.com/layers/pulsar/bewaremypower/pulsar/2.8.4/images/sha256-fba51a75c0f2ca79fbff7b254f80f641fcda661fd702f8149bbfdd5994078e3a > >> > >> > >> > https://hub.docker.com/layers/pulsar-all/bewaremypower/pulsar-all/2.8.4/images/sha256-42d4b41e5869edc6242bb49d6a1687bd6d191a6385637122edc5c7b2c44ee46f > >> > >> Please download the source package, and follow the Release Candidate > >> Validation[1] to validate the release > >> > >> [1] https://github.com/apache/pulsar/wiki/Release-Candidate-Validation > >> > >> Thanks, > >> Yunze > >> > >> > >> > >> > >> > > > > -- > > BR, > > Qiang Huang > > -- BR, Qiang Huang
Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic
Good idea. I suggest the naming endings with `Seconds` like sendDelayOfTwoPhaseDeletionInSeconds,` reconsumeLaterOfTwoPhaseDeletionInSeconds`. >private int sendDelaySecondsOfTwoPhaseDeletion = 60; >private int reconsumeLaterSecondsOfTwoPhaseDeletion = 600; Yan Zhao 于2022年8月12日周五 10:45写道： > > I suggest to include 'topic' in the flag, we have too many entities in > > Pulsar > > Thanks, change it to `topicTwoPhaseDeletionEnabled`. > -- BR, Qiang Huang
Re: [DISCUSS] Skip unnecessary tests when there are only cpp/python related changes
+1. It can help reduce a lot of useless duplication of test cases. Michael Marshall 于2022年8月11日周四 09:58写道： > Great suggestion, +1. > > - Michael > > On Wed, Aug 10, 2022 at 8:38 PM Dave Fisher wrote: > > > > Yes, please! > > > > Sent from my iPhone > > > > > On Aug 10, 2022, at 5:39 PM, PengHui Li wrote: > > > > > > +1 > > > > > > Best, > > > Penghui > > > > > >> On Wed, Aug 10, 2022 at 10:52 PM Yunze Xu > > > >> wrote: > > >> > > >> LGTM > > >> > > >> Thanks, > > >> Yunze > > >> > > >> > > >> > > >> > > >>> 2022年8月10日 15:36，Zike Yang 写道： > > >>> > > >>> Hi, Pulsar community > > >>> > > >>> Currently, Java tests consume significant CI resources. And it is not > > >>> necessary to run all the tests for changes that are only on the C++ > or > > >>> python parts of the code. I have created a PR [0] to improve the CI > by > > >>> skipping unnecessary tests when there are only CPP/Python changes. > > >>> This can significantly increase the efficiency of CI when testing the > > >>> C++/Python part of the code. > > >>> > > >>> After this PR gets merged, we will skip java unit tests, integration > > >>> tests(the part only for java codes), and go function tests when there > > >>> are only cpp/python changes. But the system test is not skipped > > >>> because there are some python function codes in that test. Perhaps in > > >>> the future, we can further optimize the system test to skip > > >>> unnecessary matrix tests for PRs with only C++ changes. > > >>> > > >>> I have created a test PR in a separate repo to verify this PR. [1] > > >>> And more detail in [2]. > > >>> > > >>> Please take a look and feel free to comment on it. > > >>> > > >>> Regarding the current Pulsar CI, I have a question. Why do we need to > > >>> add doc_only check at each step when skipping code tests instead of > > >>> just skipping the whole job for PR with only doc changes? [3] Is > there > > >>> any concern? > > >>> > > >>> Please let me know what you think. Thanks! > > >>> > > >>> > > >>> [0] https://github.com/apache/pulsar/pull/16988 > > >>> [1] https://github.com/RobertIndie/pulsar-ci-test/pull/1 > > >>> [2] > > >> https://github.com/RobertIndie/pulsar-ci-test/actions/runs/2829525510 > > >>> [3] > > >> > https://github.com/apache/pulsar/blob/master/.github/workflows/pulsar-ci.yaml#L380 > > >>> > > >>> Best, > > >>> Zike Yang > > >> > > >> > > > -- BR, Qiang Huang
Re: [Discuss] PIP 198: Standardize PR Naming Convention using GitHub Actions
I agree that the customized one is better. +1 on the customized one. Jun M 于2022年8月12日周五 10:51写道： > +1 on the customized one. > > > Cheers > momo-jun > > -- BR, Qiang Huang
Re: [DISCUSS] Switch to the Temurin JDK in the Docker image
LGTM. They are very similar. The OpenJDK provides source-code, and the Temurin JDK provides builds of the source code. I would like to point out that they have different licenses. - The OpenJDK implementation is licensed under the GPL-2.0-only with a linking exception. [0] - The Eclipse Temurin™ project provides code and processes that support the building of runtime binaries and associated technologies that are high performance, enterprise-caliber, cross-platform, open-source licensed, and Java SE TCK-tested for general use across the Java ecosystem.[1] - [0] https://en.wikipedia.org/wiki/OpenJDK https://openjdk.org/legal/ - [1] https://projects.eclipse.org/projects/adoptium.temurin Zixuan Liu 于2022年8月12日周五 15:46写道： > Hi tison, > > Great catch! > > The Temurin JDK is OpenJDK distribution from Adoptium, the old JDK from > Ubuntu, they should all be built on the OpenJDK open source project, so I > think should be fully compatible. > > Each Temurin release has passed the relevant Oracle Java Compatibility Kit > (JCK) to demonstrate that it is a compatible implementation of the Java > specification. In addition, Temurin releases must pass the AQAvit quality > verification suite[1] to ensure they are ready for production usage. > > [1] - https://adoptium.net/docs/qvs-policy > > Thanks, > Zixuan > > > tison 于2022年8月12日周五 15:23写道： > > > +1 > > > > Thanks for bringing this topic :) > > > > In Pulsar usages, these two distributions should not be quite different. > > Did you investigate the compatibility more? > > > > Best, > > tison. > > > > > > Zixuan Liu 于2022年8月12日周五 15:17写道： > > > > > Hi all, > > > > > > I noticed we are using OpenJDK in our Docker image, I suggest that we > > > switch to the Temurin JDK, because our CI runs on the Temurin JDK, we > > need > > > to keep the same JDK everywhere to avoid unexpected problems. Thanks, > > > Zixuan > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Create a new Github Project to track the flaky tests
+1. Flaky tests failures bother contributors a lot. Resolving flak test failures can greatly improve development efficiency. PengHui Li 于2022年8月13日周六 10:06写道： > I thought maybe we can close the flaky test with the Stale label. > All of them are created very early and have no recurrence within a month. > Maybe fixed by some PRs, but not sure about it. > > Currently, I haven't added them to the Github Project. > But still open in the Github issue list. > > Another one is the add-to-project CI does not work for classic projects. > Lishen https://github.com/yaalsn is trying to improve the > pulsar-test-infra > to > support adding issues/PRs to classic projects automatically. > > Best, > Penghui > > On Sat, Aug 13, 2022 at 1:50 AM Lari Hotari wrote: > > > Thank you Penghui, really useful way to coordinate the fixing of flaky > > tests! > > > > -Lari > > > > On Wed, Aug 10, 2022 at 11:35 AM PengHui Li wrote: > > > > > Hi all, > > > > > > For better tracking flaky test fix, I have tried to create a Github > > > Project under the Pulsar repo > > https://github.com/apache/pulsar/projects/11 > > > (It can be removed if we don't like this way) > > > This will help us to have a more intuitive view of all the flaky tests, > > > how many are in progress, in the review stage, and approved. > > > > > > The project is public for all the contributors, so if you want > > > to contribute some flaky tests fixes, > > > you can go to the Github Project to peek up items in the Todo column. > > > > > > And I also created a PR https://github.com/apache/pulsar/pull/17038 to > > > add the PRs and issues > > > with `flaky-tests` label to this project automatically. > > > > > > BTW, I also have some questions about the Github Project automation. As > > > the description of > > > column `Review in progress, it said the PR with request changes would > go > > > to this column > > > automatically. But it doesn't work. I'm not sure why. > > > [image: image.png] > > > > > > Best, > > > Penghui > > > > > > > > > -- BR, Qiang Huang
[jira] [Commented] (PULSAR-22) flink消费pulsar时出现错误，但是仍可消费到数据
[
https://issues.apache.org/jira/browse/PULSAR-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579311#comment-17579311
]
Qiang Huang commented on PULSAR-22:
---
Could you please submit an issue in [https://github.com/apache/pulsar/issues] ?
Also please provide a reproduce step, thank you.
> flink消费pulsar时出现错误，但是仍可消费到数据
>
>
> Key: PULSAR-22
> URL: https://issues.apache.org/jira/browse/PULSAR-22
> Project: Pulsar
> Issue Type: Bug
> Environment: kafka(json) -> flink(Stream) -> custom
> Serializer -> kop -> pulsar -> flink(stream) keyshared consum(问题所在区域)
>Reporter: 海洋饼干
>Priority: Major
>
> [Source Data Fetcher for Source: kafkaSource (4/4)#0] ERROR
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderBase
> - Error in polling message from pulsar consumer.
> java.util.concurrent.ExecutionException:
> org.apache.pulsar.client.api.PulsarClientException$TransactionConflictException:
>
> \{"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException:
>
> [persistent://public/default/HB0002_rdwBaseCtr_aaamore5005-test-pulsar-partition-0][flink-source]
> Transaction:(1,12) try to ack batch message:810:24 in pending ack
> status.","reqId":4357026755663389273,
> "remote":"tk01-bd-test-pulsar-7-139/192.168.7.139:6650",
> "local":"/192.168.34.54:9756"}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarUnorderedPartitionSplitReader.pollMessage(PulsarUnorderedPartitionSplitReader.java:98)
> at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderBase.fetch(PulsarPartitionSplitReaderBase.java:110)
> at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarUnorderedPartitionSplitReader.fetch(PulsarUnorderedPartitionSplitReader.java:56)
> at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by:
> org.apache.pulsar.client.api.PulsarClientException$TransactionConflictException:
>
> \{"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException:
>
> [persistent://public/default/HB0002_rdwBaseCtr_aaamore5005-test-pulsar-partition-0][flink-source]
> Transaction:(1,12) try to ack batch message:810:24 in pending ack
> status.","reqId":4357026755663389273,
> "remote":"tk01-bd-test-pulsar-7-139/192.168.7.139:6650",
> "local":"/192.168.34.54:9756"}
> at
> org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:1177)
> at
> org.apache.pulsar.client.impl.ClientCnx.handleAckResponse(ClientCnx.java:431)
> at
> org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:150)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
> at
> org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apac
Re: [VOTE] Pulsar Release 2.7.5 Candidate 2
+1 (non-binding) Haiting Jiang 于2022年8月13日周六 00:27写道： > Here is the docker images: > > > https://hub.docker.com/layers/271293864/jason918/pulsar/2.7.5/images/sha256-8863429ae891cfdac609455992105427188ccadef98723819c882644084748c9 > > > https://hub.docker.com/layers/271294049/jason918/pulsar-all/2.7.5/images/sha256-db90c52962abd7314973242e272b91be36c1ac34a206db482575fc65d9f3abaf > > Thanks, > Haiting > > On 2022/08/12 07:50:43 Haiting Jiang wrote: > > This is the second release candidate for Apache Pulsar, version 2.7.5. > > > > It fixes the following issues: > > https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-2 > > > > *** Please download, test and vote on this release. This vote will stay > open > > for at least 72 hours *** > > > > Note that we are voting upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-2/ > > > > SHA-512 checksums: > > > > > 50ab0c3fd954fd1911c84917be411302e8b4dd1afca8b6af67df6d4f4f74ea43125dd18c2bc84aa62b635ff3ca21c54d59085a89ef029d15e7f078854958edb5 > > apache-pulsar-2.7.5-bin.tar.gz > > > 9d8a327a728b917513b492cda585c11423705261e7f6a13d1c9e33a991b4e6f60b9aae7179a24c56e1c7bf0136d9fb64199e8b7952f2a63e8ed98bb518a05988 > > apache-pulsar-2.7.5-src.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1171/ > > > > The tag to be voted upon: > > v2.7.5-candidate-2 (43a8436ca6cd6604cd25174d7388390bcd5d6b12) > > https://github.com/apache/pulsar/releases/tag/v2.7.5-candidate-2 > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Please download the source package, and follow the README to build > > and run the Pulsar standalone service. > > > > Thanks, > > Haiting > > > -- BR, Qiang Huang
Re: [VOTE] Pulsar Release 2.8.4 Candidate 1
+1 (non-binding) Is 2.8.4 a long term support release? Yunze Xu 于2022年8月12日周五 16:20写道： > This is the first release candidate for Apache Pulsar, version 2.8.4. > > It fixes the following issues: > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.8.4 > > *** Please download, test and vote on this release. This vote will stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.4-candidate-1/ > > SHA-512 checksums: > > c3d26704f2cfb3365c29d4110612ca7351084f8bee3c306d5e906b3f9b22c7557cc5baf12f74f8c222baccae3310691419eda5b47fdf9cd6c5281b70134ab5eb > apache-pulsar-2.8.4-bin.tar.gz > 28160ee94dccfb74dfb56e0e5d0e08870c6612659507333a52b5660ecbf060a75d1eed667cffd8596f9468de95055b78916b932db0e0d4c2745868d55429ee98 > apache-pulsar-2.8.4-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1170/ > > The tag to be voted upon: > v2.8.4-candidate-1 (02ee5616866d4eda8dd94f85d9d9b71c459f248d) > https://github.com/apache/pulsar/releases/tag/v2.8.4-candidate-1 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Docker images: > > > https://hub.docker.com/layers/pulsar/bewaremypower/pulsar/2.8.4/images/sha256-fba51a75c0f2ca79fbff7b254f80f641fcda661fd702f8149bbfdd5994078e3a > > > https://hub.docker.com/layers/pulsar-all/bewaremypower/pulsar-all/2.8.4/images/sha256-42d4b41e5869edc6242bb49d6a1687bd6d191a6385637122edc5c7b2c44ee46f > > Please download the source package, and follow the Release Candidate > Validation[1] to validate the release > > [1] https://github.com/apache/pulsar/wiki/Release-Candidate-Validation > > Thanks, > Yunze > > > > > -- BR, Qiang Huang
Re: [Vote] PIP-192 New Pulsar Broker Load Balancer
+1 (non-binding) Kai Wang 于2022年8月5日周五 10:18写道： > +1 (non-binding) > > Thanks, > Kai > > Heesung Sohn 于2022年8月2日周二 08:50写道： > > > Dear Pulsar Community, > > > > Please review and vote on this PIP. > > > > PIP link: https://github.com/apache/pulsar/issues/16691 > > > > Thank you, > > -Heesung > > > -- BR, Qiang Huang
Re: [Vote] PIP 198: Standardize PR Naming Convention using GitHub Actions
+1 (non-binding) I like the idea of check PR title as a job. Good job, Yu. tison 于2022年8月8日周一 22:07写道： > +1 (non-binding) to the proposal itself. > > Although, we should later move the standard to our website where the whole > project can easily contribute to and follow the general contribution > process - that is, send a pull request, review, and merge. I regard current > gdoc content as a temporary container for this content. > > If this proposal gets accepted, @Yu you can create an issue for the dev doc > part and ping me. I can offer my help to write so. > > Best, > tison. > > > Zike Yang 于2022年8月8日周一 13:35写道： > > > +1 (non-binding) > > > > Thanks > > Zike Yang > > > > On Mon, Aug 8, 2022 at 1:06 PM Xiangying Meng > > wrote: > > > > > > +1(non-binding) > > > > > > yours sincerely, > > > xiangying Meng > > > > > > On Thu, Aug 4, 2022 at 4:13 PM Yu wrote: > > > > > > > Hi team, > > > > > > > > It has been 4 months since we discussed the [Guideline] Pulsar PR > > Naming > > > > Convention [1]. > > > > > > > > Nowadays, when reading the PR list [2], you’ll find more and more > > people > > > > follow and get used to this rule. > > > > > > > > It improves collaboration efficiency, that is great! > > > > > > > > This makes us think about moving the rule forward, that is, > > standardizing > > > > PR title naming using GitHub Actions, which is a more efficient way. > > > > > > > > So we'd like to start a vote on PIP 198: Standardize PR Naming > > Convention > > > > using GitHub Actions [3]. > > > > > > > > > > > > This proposal contains: > > > > > > > > - Why do this? > > > > > > > > - How do this? > > > > > > > > - Pre-discussions and other thoughts > > > > > > > > Feel free to comment, thank you! > > > > > > > > [1] https://lists.apache.org/thread/sk9ops3t94jmzc5tndk08y9khf7pj6so > > > > > > > > [2] https://github.com/apache/pulsar/pulls > > > > > > > > [3] > > > > > > > > > > > https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1# > > > > > > > > > > > > Yu, Max, mangoGoForward > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] PIP 194 : Pulsar client: seek command add epoch
Sure. You can refer to pip-84: https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch . Zike Yang 于2022年7月29日周五 10:22写道： > Hi, Qiang > > > It is necessary to check the current cursor status when handling > flowPermits > > request from the server side. If the server is handling seek request, it > > should ignore flowPermits request because the request is illegal. > > Thanks for your explanation. I think it's better to add this > explanation to the PIP. > > > The reconnected consumer can regard as a new consumer with new epoch. > > The consumer will reconnect to the broker during the seek operation. > And this will change the existing behavior. It doesn't seem to make > sense. Please correct me if I have misunderstood. > > Thanks, > Zike Yang > > On Wed, Jul 27, 2022 at 8:06 PM Qiang Huang > wrote: > > > > Thanks Zike. > > > > - stage 1: Check the current cursor status when handling flowPermits > > from > > > > the server side. > > > > > > Could you explain more details on this step? It looks like there is > > not much described above. What kind of status needs to be checked, and > > what kind of behavior will the broker take? > > It is necessary to check the current cursor status when handling > flowPermits > > request from the server side. If the server is handling seek request, it > > should ignore flowPermits request because the request is illegal. > > > > > > > > 1. Consumer reconnect need reset epoch. > > >> Why do we need to reset the epoch when the consumer reconnects? > > The reconnected consumer can regard as a new consumer with new epoch. > -- BR, Qiang Huang
Re: [VOTE] PIP-180: Shadow Topic, an alternative way to support readonly topic ownership
+1(non-binding) Haiting Jiang 于2022年8月1日周一 10:23写道： > Dear Community, > > I would like to restart the VOTE on "PIP-180: Shadow Topic, an alternative > way to support readonly topic ownership." > > The proposal is updated since last vote and all can be read at [0] and the > discussion thread is available at [1]. > Voting will stay open for at least 48h. > > [0] https://github.com/apache/pulsar/issues/16153 > [1] https://lists.apache.org/thread/o9k7trfmxrz89b0woybnshonpkq8ybw1 > > Thanks, > Haiting Jiang > -- BR, Qiang Huang
Re: [DISCUSS] PIP 194 : Pulsar client: seek command add epoch
Thanks Zike. > > - stage 1: Check the current cursor status when handling flowPermits from > > the server side. > > Could you explain more details on this step? It looks like there is not much described above. What kind of status needs to be checked, and what kind of behavior will the broker take? It is necessary to check the current cursor status when handling flowPermits request from the server side. If the server is handling seek request, it should ignore flowPermits request because the request is illegal. > > 1. Consumer reconnect need reset epoch. >> Why do we need to reset the epoch when the consumer reconnects? The reconnected consumer can regard as a new consumer with new epoch.
Re: [ANNOUNCE] Micheal Marshall as a new PMC member in Pulsar
Congratulations, Micheal !!! Max Xu 于2022年7月27日周三 14:08写道： > Congratulations, Michael ! > > Best, > Max Xu > > > On Wed, Jul 27, 2022 at 2:04 PM Kai Wang > wrote: > > > Congratulations Michael! > > > > Thanks, > > Kai > > > -- BR, Qiang Huang
Re: [VOTE]PIP-189: No batching if only one message in batch.
+1(non-binding) BR, Qiang Huang mattison chao 于2022年7月25日周一 13:17写道： > +1(non-binding) > > Best, > Mattison > > On Mon, 25 Jul 2022 at 10:35, Haiting Jiang > wrote: > > > > +1 > > Thanks, > > Haiting > > > > On 2022/07/25 02:23:20 Anon Hxy wrote: > > > Dear Community, > > > > > > I would like to start a VOTE on "PIP-189: No batching if only one > message > > > in batch." > > > > > > The proposal can be read at [0] and the discussion thread is available > at > > > [1] and the PR link is available at [2] > > > > > > Voting will stay open for at least 48h. > > > > > > [0] https://github.com/apache/pulsar/issues/16619 > > > [1] https://lists.apache.org/thread/dbq1lrv03bhtk0lr5nwm5txo9ndjplv0 > > > [2] https://github.com/apache/pulsar/pull/16605 > > > > > > Thanks, > > > Xiaoyu > > > > -- BR, Qiang Huang
[DISCUSS] PIP 194 : Pulsar client: seek command add epoch
Hi Pulsar community:
I open a pip to discuss "Pulsar client: seek command add epoch"
Proposal Link:
- issue link: https://github.com/apache/pulsar/issues/16757
--
## Motivation
`Reader` belongs to exclusive subscription type, and it uses `nonDurable`
cursor. After receiving messages, `Reader` will ack cumulatively
immediately.
The `flowPermits` are triggered in multiple scenarios from the client side
and it is isolated from `seek` of `Consumer`. Therefore, it is possibile
that `flowPermits` will execute after `seek` from the client side, like the
following flow chart.
[image: image.png]
When `handleSeek` processing is delay from the server side, the `MarkDelete
position` is modified in a wrong way.
The expected result is that `Reader`can re-consume messages from `mark
delete:(1,1)` after `seek`. But it doesn't work.
Pulsar read message and seek position is not a synchronous operation, the
seek request can't prevent an in-process entry reading operation. The
client-side also has an opportunity to receive messages after the seek
position.
Pulsar client make read messages operation and seek position operation
synchronized so add an epoch into server and client consumer. After client
reader consumer invoke `seek` , the epoch increase 1 and send `seek`
command carry the epoch and then server consumer will update the epoch.
When dispatcher messages to client will carry the epoch which the cursor
read at the time. Client consumer will filter the send messages command
which is smaller than current epoch.
In this way, after the client consumer send `seek` command successfully,
because it has passed the epoch filtering, the consumer will not receive a
message with a messageID greater than the user previously seek position.
### Current implementation details
CommandSeek Protocal
```proto
// Reset an existing consumer to a particular message id
message CommandSeek {
required uint64 consumer_id = 1;
required uint64 request_id = 2;
optional MessageIdData message_id = 3;
optional uint64 message_publish_time = 4;
}
```
### CommandMessage
```proto
message CommandMessage {
required uint64 consumer_id = 1;
required MessageIdData message_id = 2;
optional uint32 redelivery_count = 3 [default = 0];
repeated int64 ack_set = 4;
optional uint64 epoch = 5 [default = 0];
}
```
`CommandMessage` already add epoch by [PIP-84](
https://github.com/apache/pulsar/wiki/PIP-84-:-Pulsar-client:-Redeliver-command-add-epoch)
, when client receive `CommandMessage` will compare the command epoch and
local epoch to handle this command.
## Goal
Add epoch into seek command.
## API Changes
### Protocal change: CommandSeek
```proto
// Reset an existing consumer to a particular message id
message CommandSeek {
required uint64 consumer_id = 1;
required uint64 request_id = 2;
optional MessageIdData message_id = 3;
optional uint64 message_publish_time = 4;
optional uint64 consumer_epoch = 5;
}
```
`CommandSeek` command add epoch field, when client send seek command to
server successfully, the server will change the server consumer epoch to
the command epoch. The epoch only can bigger than the old epoch in server.
Now the client can filter out the message which contains less consumer
epoch.
## Implementation
- stage 1: Check the current cursor status when handling flowPermits from
the server side.
- stage 2: Add epoch into seek command, and server update the consumer
epoch. It can prevent an in-process entry reading operation after the seek
request.
## Reject Alternatives
None yet.
## Note
1. Consumer reconnect need reset epoch.
--
BR,
Qiang Huang
Re: [DISCUSS] PIP-191: Support batched message using entry filter
Good. +1
Anon Hxy 于2022年7月19日周二 18:00写道：
> Hi Pulsar community:
>
> I open a pip to discuss "Support batched message using entry filter"
>
> Proposal Link: https://github.com/apache/pulsar/issues/16680
> ---
>
> ## Motivation
>
> We already have a plug-in way to filter entries in broker, aka PIP-105
> https://github.com/apache/pulsar/issues/12269. But this way only checks
> at
> the batch header, without digging into the individual messages properties.
> Of course it provides an interface to deserialize the entire Entry to the
> heap, But this will bring some memory and cpu workload. And in most
> scenarios we only need partial properties to do some filter.
>
> This proposal brings a method to make PIP-105 support batched entry without
> having to deserialize the entire Entry to the heap
>
>
> ## API Changes
>
> - Add a producer config to specialize the key, of which properties will be
> added to the batched entry metadata, for example:
>
> ```
>
> org.apache.pulsar.client.impl.conf.ProducerConfigurationData#batchedFilterProperties
>
> ```
> The `batchedFilterProperties` type is `List` with default value is
> empty list. For an empty list, it means that the properties of entry's
> metadata are empty, and the `EntryFilter` will not take effect.
>
> ## Implementation
>
> - When call `org.apache.pulsar.client.impl.BatchMessageContainerImpl#add`,
> we extract the message properties and add it to `metadata`:
> ```
> public boolean add(MessageImpl msg, SendCallback callback) {
>
> if (log.isDebugEnabled()) {
> log.debug("[{}] [{}] add message to batch, num messages in
> batch so far {}", topicName, producerName,
> numMessagesInBatch);
> }
>
> if (++numMessagesInBatch == 1) {
> try {
> // some properties are common amongst the different
> messages in the batch, hence we just pick it up from
> // the first message
> messageMetadata.setSequenceId(msg.getSequenceId());
> List filterProperties = getProperties(msg);
> if (!filterProperties.isEmpty()) {
> messageMetadata.addAllProperties(filterProperties); //
> and message properties here
> }
> ```
>
> - Also we need to add a method `hasSameProperties` like `hasSameSchema`.
> Messages with same properties can be added to the same batch:
>
> ```
> private boolean canAddToCurrentBatch(MessageImpl msg) {
> return batchMessageContainer.haveEnoughSpace(msg)
>&& (!isMultiSchemaEnabled(false)
> || batchMessageContainer.hasSameSchema(msg)
> || batchMessageContainer.hasSameProperties(msg))
> && batchMessageContainer.hasSameTxn(msg);
> }
>
> ```
>
>
> ## Reject Alternatives
>
> - Implement a `AbstractBatchMessageContainer` , saying
> `BatchMessagePropertiesBasedContainer`, keeping messages with same
> properties in a single `hashmap` entry, like
> `BatchMessageKeyBasedContainer`.
>
> Rejection reason: This will publish messages out of order
>
>
>
> Thanks,
> Xiaoyu Hou
>
--
BR,
Qiang Huang
Re: [VOTE] Enable GitHub Discussion for user-facing discussion
+1 guo jiwei 于2022年7月13日周三 10:11写道： > +1 > > > Regards > Jiwei Guo (Tboy) > > > On Tue, Jul 12, 2022 at 9:53 PM Enrico Olivelli > wrote: > > > +1 > > > > > > Enrico > > > > Il giorno mar 12 lug 2022 alle ore 13:05 Yu ha > scritto: > > > > > > Hi tison, thanks for creating this vote! > > > I thought we reached a lay consensus since there was no objection since > > the > > > last discussion [1] > > > > > > +1 for this proposal. > > > I've explained the reasons and showed the benefits previously [2] > > > > > > [1] https://lists.apache.org/thread/1y7zbchlbokwnpd0jv2tt5jtzg6px6yn > > > [2] https://lists.apache.org/thread/83pst643h9cqcryo3zsjd240jmqzvn73 > > > > > > On Tue, Jul 12, 2022 at 6:01 PM tison wrote: > > > > > > > Hi, > > > > > > > > In the previous email[1] I started a discussion about enabling GitHub > > > > Discussions in apache/pulsar repository and we meet a consensus to > > avoid > > > > making development discussions truth happen on the sources. It's > also a > > > > requirement of INFRA team[2] > > > > > > > > Apart of it, for user-facing discussions it's conventions for Pulsar > > users > > > > who have a GitHub account but are unfamiliar with mailing list to > throw > > > > their use case and questions on GitHub discussion. > > > > > > > > Dave has opened and issue[2][3] and prepare the patch[4] for turning > > on the > > > > discussion option but obviously we still need an explicit consensus > > among > > > > the community and it's also required before INFRA team takes action. > > > > > > > > I'd like to start this voting thread for "Enable GitHub Discussion > for > > > > user-facing discussion", please reply with your opinion: > > > > > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove (and reason why) > > > > > > > > Best, > > > > tison. > > > > > > > > [1] https://lists.apache.org/thread/1y7zbchlbokwnpd0jv2tt5jtzg6px6yn > > > > [2] https://issues.apache.org/jira/browse/INFRA-23477 > > > > [3] https://github.com/apache/pulsar/issues/16275 > > > > [4] https://github.com/apache/pulsar/pull/16528 > > > > > > > -- BR, Qiang Huang
Re: Batch Messages with only 1 message
+1 Good idea! It will greatly reduce the resource consumption of small batches. Yubiao Feng 于2022年7月12日周二 16:00写道： > +1 > Good idea. > > Thanks > Yubiao Feng > > On Tue, Jul 12, 2022 at 3:54 PM Enrico Olivelli > wrote: > > > Hello, > > I think that we could implement a small but effective enhancement to > > batching. > > > > It may happen that even if you enable batching you come to create > > entries with 1 only message. > > > > Processing batch messages requires a good amount of resources, both on > > the broker and on the client side. > > > > Especially when you are using PIP-105, Broker side filtering, you have > > to unpack (and decompress) the whole entry in order to process the > > very single message. > > > > So my proposal is to change the (Java) producer, to make it produce a > > regular message instead of a batch message if the batch contains only > > 1 message > > > > Thoughts ? > > > > Enrico > > > -- BR, Qiang Huang
Re: [ANNOUNCE] New Committer: Zixuan Liu
Congratulations!!! Zixuan. Max Xu 于2022年7月8日周五 21:36写道： > Congratulations! Zixuan > > Best > Max Xu > > > > On Thu, Jul 7, 2022 at 8:55 PM Nicolò Boschi wrote: > > > Congrats! > > > > Nicolò Boschi > > > > > > Il giorno gio 7 lug 2022 alle ore 13:28 Haiting Jiang < > > jianghait...@apache.org> ha scritto: > > > > > Congratulates, Zixuan! > > > > > > BR, > > > Haiting > > > > > > On 2022/07/07 10:03:36 Yu wrote: > > > > Hi team, > > > > > > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > > > Zixuan Liu (https://github.com/nodece) to become a committer > > > > and we are pleased to announce that he has accepted. > > > > > > > > Being a committer enables easier contribution to the > > > > project since there is no need to go via the patch > > > > submission process. This should enable better productivity. > > > > > > > > Welcome and congratulations, Zixuan Liu! > > > > > > > > Please join us in congratulating and welcoming Zixuan Liu onboard! > > > > > > > > Best Regards, > > > > Yu on behalf of the Pulsar PMC > > > > > > > > > > -- BR, Qiang Huang
Re: [VOTE] Pulsar Release 2.9.3 Candidate 2
+1 Enrico Olivelli 于2022年7月8日周五 19:42写道： > +1 (binding) > - built from sources, with JDK11, run a couple of smoke tests > - verified RAT > - verified checksums and signatures > - run some JMS tests using Transactions (the JMS library bundles > 2.10.x Java client) > > Thanks for driving the release! > > Enrico > > Enrico > > Il giorno mer 6 lug 2022 alle ore 10:26 mattison chao > ha scritto: > > > > This is the second release candidate for Apache Pulsar, version 2.9.3. > > > > It fixes the following issues: > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.9.3+ > < > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.9.3+ > > > > > > *** Please download, test and vote on this release. This vote will stay > open > > for at least 72 hours *** > > > > Note that we are voting upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.3-candidate-2/ > > > > SHA-512 checksums: > > > > > 291eb3f9da234cf38fcd02de781def9a9354025bb4f98c78b160935a6a9c6721cc8280d80b93049656ee5a20e36ddc5d3446b7b034405c07d447833ff65e > ./apache-pulsar-2.9.3-bin.tar.gz > > > > > d57fa3c8eae1f3ba60422a56288c99a472a671295e41573c884a9d9a71b5fcf622782732e9cfd5128e1b92304b3812cc877675384ac0dbc78109d7efb23681f4 > ./apache-pulsar-2.9.3-src.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1163/ > > > > The tag to be voted upon: > > v2.9.3-candidate-2 (dd9a5f1f91651b634600f66c53dcc6ad855fb669) > > > > https://github.com/apache/pulsar/releases/tag/v2.9.3-candidate-2 > > > > The docker images: > > > > > https://hub.docker.com/layers/248117318/mattison/pulsar/2.9.3-rc-2/images/sha256-a7ac6d5ffb2d77102ca6633313c9d0265c1c89c8b0fe859023fbaf3e0d0a7910?context=repo > > > > > https://hub.docker.com/layers/247907545/mattison/pulsar-all/2.9.3-rc-2/images/sha256-00e6a886a9285107027afb4c3218c9340efb96627100ff2f1c95d1177bd8dbbe?context=repo > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS < > https://dist.apache.org/repos/dist/dev/pulsar/KEYS> > > > > Please download the source package, and follow the Release Candidate > > Validation[1] > > to validate the release > > > > [1] https://github.com/apache/pulsar/wiki/Release-Candidate-Validation < > https://github.com/apache/pulsar/wiki/Release-Candidate-Validation> > -- BR, Qiang Huang
Re: [DISSCUSS]PIP-181: Reduce unnecessary REST call in broker
().getAdminClient().topics().getStatsAsync(
> >>>>
> >>>> (topicName.getPartition(i).toString()), getPreciseBacklog,
> >>>> subscriptionBacklogSize,
> >>>>getEarliestTimeInBacklog));
> >>>>} catch (PulsarServerException e) {
> >>>>asyncResponse.resume(new RestException(e));
> >>>>return;
> >>>>}
> >>>>}
> >>>> ```
> >>>>
> >>>> * Suggest to do like this:
> >>>> ```
> >>>> for (int i = 0; i < partitionMetadata.partitions; i++) {
> >>>>TopicName topicNamePartition =
> topicName.getPartition(i);
> >>>>topicStatsFutureList.add(
> >>>>
> >>>> pulsar().getNamespaceService().isServiceUnitOwnedAsync(topicName)
> >>>>.thenCompose(owned -> {
> >>>>if (owned) {
> >>>>// local call
> >>>>return
> >>>> getTopicReferenceAsync(topicNamePartition)
> >>>>.thenCompose(topic ->
> >>>>
> >>>> topic.asyncGetStats(getPreciseBacklog, subscriptionBacklogSize,
> >>>>getEarliestTimeInBacklog));
> >>>>} else {
> >>>>// call from admin client
> >>>>try {
> >>>>
> >>>>
> pulsar().getAdminClient().topics().getStatsAsync(topicNamePartition.toString()),
> >>>>getPreciseBacklog,
> >>>> subscriptionBacklogSize, getEarliestTimeInBacklog)
> >>>>} catch (PulsarServerException e) {
> >>>>throw new RestException(e);
> >>>>}
> >>>>}
> >>>>})
> >>>>);
> >>>> ```
> >>>>
> >>>> Thanks,
> >>>> Xiaoyu Hou
> >>>>
> >>>>
> >>>>
>
>
--
BR,
Qiang Huang
[VOTE] [PIP-182] Provide new load balance placement strategy implementation for ModularLoadManagerStrategy
Hi Pulsar Community, I would like to start a VOTE on "Provide new load balance placement strategy implementation for ModularLoadManagerStrategy" (PIP-182). The proposal can be read at https://github.com/apache/pulsar/issues/16274 . The PR link is https://github.com/apache/pulsar/pull/16281 and the discussion thread is available at https://lists.apache.org/thread/36vyyhndr4og175k2bz3mfdf5ctd2xky Voting will stay open for at least 48h. -- BR, Qiang Huang
Re: [DISCUSS] PIP-181: Provide new load balance placement strategy implementation for ModularLoadManagerStrategy
servations. > > > > How to calculate a score for a broker ? > > - use its historical load and short-term load data with weight. > > > > How to select a broker for assignning bundle ? > > - select a broker based on which one has the least resource usage with > > weight. > > > > ### New configuration options > > The existing cache implementation will not be removed at this point. > Users > > will > > be able to configure the old implementation in `broker.conf`. > > This option will be helpful in case of performance regressions would be > > seen for > > some use cases with the new strategy implementation. > > ``` > > # load assignment strategy, support LeastLongTermMessageRate and > > LeastResourceUsageWithWeight, default is LeastLongTermMessageRate > > > > > loadBalancerLoadAssignmentStrategy=org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight > > ``` > > > > Below are screenshots of the effect of the new strategy with less time > and > > fewer load balancing times. > > https://user-images.githubusercontent.com/4970972/176346492-f2ccdfda-b011-406d-88fe-df73d8bb839b.png > > "> > > https://user-images.githubusercontent.com/4970972/176346531-63a9b8b0-ef7b-4f74-a904-37d7c07c1793.png > > "> > > > > ## Reject Alternatives > > None yet. > > > > ## Reference > > [1] https://github.com/apache/pulsar/pull/16059 > > [2] https://github.com/apache/pulsar/issues/16274 > > [3] https://github.com/apache/pulsar/pull/16281 > > > > -- > > BR, > > Qiang Huang > > > -- BR, Qiang Huang
Re: [VOTE] [PIP-179] Support the admin API to check unknown request parameters
+1 Yubiao Feng 于2022年6月30日周四 00:40写道： > Hi Pulsar Community > > I would like to start a VOTE on "Support the admin API to check unknown > request parameters" (PIP-179). > > The proposal can be read at https://github.com/apache/pulsar/issues/16135 > > and the discussion thread is available at > https://lists.apache.org/thread/m8vkxl46njm7sh0r1mqsn25jggq9v8kb > > Voting will stay open for at least 48h. > > Thanks > Yubiao Feng > -- BR, Qiang Huang
