[VOTE] Pulsar Client Go Release 0.13.0 Candidate 2
Hi everyone, Please review and vote on the release candidate #2 for the version 0.13.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the second release candidate for Apache Pulsar Go client, version 0.13.0. It fixes the following issues: https://github.com/apache/pulsar-client-go/milestone/15?closed=1 Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://downloads.apache.org/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1245 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.13.0-candidate-2/ The tag to be voted upon: v0.13.0-candidate-2 https://github.com/apache/pulsar-client-go/tree/v0.13.0-candidate-2 SHA-512 checksums: 9a52fb25cbb6d86651a0d56c1d6e17826810b91f3ba6299f632630f5b5a1d85c6a0842e36aaa2da6fc50d4e9406fe6d7b557368f9d99d876345c987f51d554fb apache-pulsar-client-go-0.13.0-src.tar.gz
[VOTE] Pulsar Client Go Release 0.13.0 Candidate 1
Hi everyone, Please review and vote on the release candidate #1 for the version 0.13.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the first release candidate for Apache Pulsar Go client, version 0.13.0. It fixes the following issues: https://github.com/apache/pulsar-client-go/milestone/15?closed=1 Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://downloads.apache.org/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1245 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.13.0-candidate-1/ The tag to be voted upon: v0.13.0-candidate-1 https://github.com/apache/pulsar-client-go/tree/v0.13.0-candidate-1 SHA-512 checksums: 05cf45eea4e2543763f4de281530df6377dbfd014ea57d043d944cdb09ea84ffbbbdda7ce86ce892f9ae0f88174179684d94fde29a91ca2d286c3c6c3e248545 apache-pulsar-client-go-0.13.0-src.tar.gz
Re: [VOTE] Pulsar Node.js Client Release 1.11.1 Candidate 2
+1 (binding) - Verified checksum and signature - Built from source on macOS arm64 and run the example - Installed from the npm registry using Node 16,18,20 on Ubuntu 22.04 amd64 and macOS arm64 - Ran the end-to-end example successfully Thanks, Zike Yang On Mon, Jul 8, 2024 at 9:19 AM Baodi Shi wrote: > > +1(binding) > > - Verified checksum and signatures > - Built from source and ran examples on macOS with Node.js v20.4.0 > - Verify batch_receive API on ubuntu 22.04 > > Thanks, > Baodi Shi > > Yunze Xu 于2024年7月5日周五 15:43写道： > > > > +1 (binding) > > > > - Verified checksum and signatures > > - Built from source and ran examples on macOS with Node.js v20.4.0 > > - Install the npm and ran examples on Rocky Linux 8 with Node.js v16.6.0 > > > > Thanks, > > Yunze > > > > On Wed, Jul 3, 2024 at 10:01 AM Baodi Shi wrote: > > > > > > Hi everyone, > > > > > > This is the second release candidate for Apache Pulsar Node.js client, > > > version 1.11.1. > > > (Candidate 1 build failed, so skip voting for it.) > > > > > > It fixes the following issues: > > > - > > > https://github.com/apache/pulsar-client-node/compare/v1.11.0...v1.11.1-rc.2 > > > > > > Please download the source files and review this release candidate: > > > - Download the source package, verify shasum and asc > > > - Follow the README.md to build and run the Pulsar Node.js client. > > > > > > The release candidate package has been published to the npm registry: > > > https://www.npmjs.com/package/pulsar-client/v/1.11.1-rc.2 > > > > > > You can install it and verify the package: > > > "npm i pulsar-client@1.11.1-rc.2 > > > --pulsar_binary_host_mirror=https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/; > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Source files: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.11.1-rc.2/ > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > > https://downloads.apache.org/pulsar/KEYS > > > > > > SHA-512 checksum: > > > 2c6a73502cb96e7e9bae958dac14ec98eed9131e3ec9796c07e4095f269f6c33df0fef32aa2bea810f321a708e5184c7526e6bbf803a2f954823ad55612623d2 > > > ./apache-pulsar-client-node-1.11.1.tar.gz > > > > > > > > > The tag to be voted upon: > > > v1.11.1-rc.2 > > > https://github.com/apache/pulsar-client-node/releases/tag/v1.11.1-rc.2 > > > > > > Please review and vote on the release candidate #1 for the version > > > 1.11.1, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > Thanks, > > > Baodi Shi
Re: [DISCUSS] Release Pulsar Go Client 0.13.0
> But why do I find that some PRs have already been released in version v0.12.1? Yes. This change list(https://github.com/apache/pulsar-client-go/compare/v0.12.1...master) is inconsistent. Some PRs have already been released. I have updated the milestone for version 0.13.0. And this should be the change list for the 0.13.0: https://github.com/apache/pulsar-client-go/milestone/15?closed=1 Thanks, Zike Yang On Wed, Jun 26, 2024 at 2:21 PM Jie crossover wrote: > > +1 > > But why do I find that some PRs have already been released in version > v0.12.1? > https://github.com/apache/pulsar-client-go/pull/1156 > -- > Best Regards！ > crossoverJie > > > Zike Yang 于2024年6月26日周三 10:16写道： > > > Hi everyone, > > > > I would like to propose releasing the Pulsar Go Client 0.13.0. > > > > It has been several months since the last release. And there are > > several new features and bug fixes in the master branch[0]. It's time > > to release a new version. > > > > Please let me know if you have any PRs that need to be included in 0.13.0. > > > > [0] https://github.com/apache/pulsar-client-go/compare/v0.12.1...master > > > > Thanks, > > Zike Yang > >
[DISCUSS] Release Pulsar Go Client 0.13.0
Hi everyone, I would like to propose releasing the Pulsar Go Client 0.13.0. It has been several months since the last release. And there are several new features and bug fixes in the master branch[0]. It's time to release a new version. Please let me know if you have any PRs that need to be included in 0.13.0. [0] https://github.com/apache/pulsar-client-go/compare/v0.12.1...master Thanks, Zike Yang
[ANNOUNCE] New Committer: Jie Chen
The Project Management Committee (PMC) for Apache Pulsar has invited Jie Chen(https://github.com/crossoverJie) to become a committer and we are pleased to announce that they have accepted. Welcome and Congratulations, Jie Chen! Please join us in congratulating and welcoming Jie Chen onboard! Best Regards, Zike Yang on behalf of the Pulsar PMC
Re: [VOTE] Release Apache Pulsar 3.3.0 based on 3.3.0-candidate-4
+1 (binding) - Verified signature and checksums - Built from source - Ran pulsar standalone - Verified the docker image with the Go client (https://github.com/RobertIndie/pulsar-client-go/pull/1) BR, Zike Yang On Wed, Jun 5, 2024 at 10:08 AM Yunze Xu wrote: > > +1 (binding) > > - Verified signature and checksums > - Verified the docker image with the C++ client > (https://github.com/BewareMyPower/pulsar-client-cpp/pull/31) > - Built from source with JDK 17 on macOS > - Verified standalone with produce and consume > > Thanks, > Yunze > > On Wed, Jun 5, 2024 at 4:28 AM Andrey Yegorov wrote: > > > > +1 (non-binding) > > > > * built from the tag > > * built from archived sources (jdk 17, mac) > > * tested pulsar standalone (jdk 17, mac) > > * checked console consumer and producer > > * checked pulsar-admin > > * spot-checked staging maven repo > > > > On Mon, Jun 3, 2024 at 3:52 AM Cong Zhao wrote: > > > > > Hello Apache Pulsar Community, > > > > > > This is a call for the vote to release the Apache Pulsar version 3.3.0 > > > based on 3.3.0-candidate-4. > > > > > > Included changes since the previous release: > > > https://github.com/apache/pulsar/milestone/38?closed=1 > > > > > > *** Please download, test and vote on this release. This vote will stay > > > open > > > for at least 72 hours *** > > > > > > Only votes from PMC members are binding, but members of the community are > > > encouraged to test the release and vote with "(non-binding)". > > > > > > Note that we are voting upon the source (tag), binaries are provided for > > > convenience. > > > > > > The release candidate is available at: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.3.0-candidate-4/ > > > > > > SHA-512 checksums: > > > > > > a19f80caa7dcc92d6df235af9f71e90d9ab52739e2d7d1ddc59944ed11be8f5f6a083db4792cafd87802d68ef9edce88e1ed91ee43cac2a98c3782b8925f07f6 > > > > > > 03692f2b6d574ce523418850d47391b9aa9f71a2bde0b593d6e1eaaf34eb6ebe080e598231d6faf2902a9f071192c0a6982403095a9374d4357b7bebb18e5e93 > > > > > > Maven staging repo: > > > https://repository.apache.org/content/repositories/orgapachepulsar-1300 > > > > > > The tag to be voted upon: > > > v3.3.0-candidate-4 (commit c0aab493aafa5386dbf93c0b58a66a666aeba17a) > > > https://github.com/apache/pulsar/releases/tag/v3.3.0-candidate-4 > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > > https://downloads.apache.org/pulsar/KEYS > > > > > > Docker images: > > > docker pull czcoder/pulsar3.3.0-c0aab49 > > > > > > https://hub.docker.com/layers/czcoder/pulsar/3.3.0-c0aab49/images/sha256-996c6babec1f887cf2f43cac0692e06f879364fe87372fcb6c8b9d3ac779c7d6?context=explore > > > docker pull czcoder/pulsar-all:3.3.0-c0aab49 > > > > > > https://hub.docker.com/layers/czcoder/pulsar-all/3.3.0-c0aab49/images/sha256-dd781dfcc72dfb3db6c0053c656dfdb750c8d36e4c57f4bbad87aee4a9cdce28?context=explore > > > > > > Please download the source package, and follow the README to build > > > and run the Pulsar standalone service. > > > > > > More advanced release validation instructions can be found at > > > https://pulsar.apache.org/contribute/validate-release-candidate/ > > > > > > Thanks, > > > Cong Zhao > > >
Re: [VOTE] PIP-356: Support Geo-Replication starts at earliest position
+1 (binding) Thanks, Zike Yang On Thu, May 30, 2024 at 3:29 PM Yubiao Feng wrote: > > Hi, all > > I would like to start the voting thread for PIP-356: Support > Geo-Replication at the earliest position. > > https://github.com/apache/pulsar/pull/22806 > > Thanks > Yubiao Feng
Re: [DISCUSS] Report an error when seeking a multi-topics consumer that subscribes no topics
I think the key point is whether we should consider a regex consumer
without any subscribed topics as a valid consumer.
> For example, when users create a regex consumer that subscribes to
topics ("tp0", "tp1", ..., "tpN") and seek to the earliest, they might
expect to consume messages from these topics. However, if they used a
wrong regex like "tp*" (it should be "tp.*), then they might find no
messages are available even if they're sure there is at least one
non-empty topic whose name starts at "tp" from the topic stats.
Actually, for this example, if we treat a consumer without any
subscribed topics as an invalid consumer, we should throw an expcetion
when the regex consumer first subscribes, instead of waiting until the
seek to inform the user that there are no topics. Currently, however,
no exception is thrown when a regex consumer subscribes without any
topics.
I prefer not to break the current behavior as it's probably not a bug.
Alternatively, we could introduce a new option for users who do not
want to seek on the consumer without any topics.
BR,
Zike Yang
On Thu, May 30, 2024 at 1:10 AM Yunze Xu wrote:
>
> For example, when users create a regex consumer that subscribes to
> topics ("tp0", "tp1", ..., "tpN") and seek to the earliest, they might
> expect to consume messages from these topics. However, if they used a
> wrong regex like "tp*" (it should be "tp.*), then they might find no
> messages are available even if they're sure there is at least one
> non-empty topic whose name starts at "tp" from the topic stats.
>
> Thanks,
> Yunze
>
> On Wed, May 29, 2024 at 7:32 PM Enrico Olivelli wrote:
> >
> > Il giorno mer 29 mag 2024 alle ore 11:10 Baodi Shi ha
> > scritto:
> >
> > > +1
> > >
> > >
> > > Thanks,
> > > Baodi Shi
> > >
> > >
> > > On May 29, 2024 at 14:50:12, Yunze Xu wrote:
> > >
> > > > Hi community,
> > > >
> > > > Recently I found the behavior of seeking a multi-topics consumer is
> > > > intuitive. If there are no internal consumers, the `seek` call will
> > > > just succeed silently.
> > >
> >
> >
> > > >
> > > > If a consumer subscribes with a regex and no topics are found, users
> > > > might think the seek operation succeeds while no topics are sought.
> > > >
> > > > I suggest throwing an exception in this case to notify users that the
> > > > consumer to seek is a multi-topics consumer that contains no topics.
> > > > It's a breaking change so I'd like to hear more voices in the mail
> > >
> >
> > What happens if it subscribed to a regex which "currently" doesn't not
> > match any topics ?
> > Maybe this is intentional in the application design as maybe you have
> > dynamic topics
> >
> > Giving an error or failing silently doesn't fix the problem.
> > If the application is sure that you must have at least N topics then we
> > should make this configurable somewhere.
> > The difference between "zero topics" or "one topic" is very small, if your
> > application expects to subscribe to 100 topics.
> >
> >
> > Enrico
> >
> >
> >
> > > > list.
> > > >
> > > > For now, I report an error for the C++ client [1], which might also
> > > > affect the Python and Node.js clients. But the behavior of the Java
> > > > client does not change.
> > > >
> > > > [1] https://github.com/apache/pulsar-client-cpp/pull/426
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > >
Re: [VOTE] Release Apache Pulsar 3.3.0 based on 3.3.0-candidate-3
+1 (binding) - Build from source - Checked the signatures - Run standalone - Run pulsar-client-go tests with image: czcoder/pulsar:3.3.0-0771f81 https://github.com/RobertIndie/pulsar-client-go/pull/1 BR, Zike Yang On Wed, May 29, 2024 at 9:25 AM PengHui Li wrote: > > +1 (binding) > > - Build from source > - Checked the signatures > - Run standalone > - Verified the Cassandra connector > - Verified the Stateful function > - Run pulsar-perf with 100 topics and batch disabled > > Regards, > Penghui > > On Wed, May 29, 2024 at 8:32 AM PengHui Li wrote: > > > Sorry for the confusion. > > After checking more details, the fix makes more sense. For more cases, > > user think it's a bug if you can grant permission to a non-existent topic. > > I will verify this release today. > > > > Thanks, > > Penghui > > > > On Tue, May 28, 2024 at 4:33 PM PengHui Li wrote: > > > >> Hi Cong, > >> > >> Thanks for driving the release. > >> This PR https://github.com/apache/pulsar/pull/22547 has changed the > >> behavior of > >> permission operations on a topic. > >> > >> The issue is it will break the user's case which relies on the lazy > >> auto-creation of the topic. > >> They can grant permission to the topic but the topic will only be created > >> if the producer or consumer is connected > >> The namespace level permission may not work because some topics need to > >> be closed down further. > >> > >> I would like to revert this change first, and then we can find a good > >> solution in the next release > >> > >> Regards, > >> Penghui > >> > >> On Mon, May 27, 2024 at 11:33 AM Cong Zhao wrote: > >> > >>> Thanks Yong, > >>> > >>> Yes, we can continue to release 3.3 and fix this issue in later versions. > >>> > >>> On 2024/05/27 03:10:18 Yong Zhang wrote: > >>> > If the snappy in zookeeper is not used very commonly, this might not > >>> be a > >>> > blocker for this release. > >>> > > >>> > So I would be -0. > >>> > > >>> > Thanks, > >>> > Yong > >>> > On Fri, 24 May 2024 at 15:17, Yong Zhang > >>> wrote: > >>> > > >>> > > -1 > >>> > > > >>> > > There is an issue with the snappy usage in the amd64-based image. > >>> > > > >>> > > It will get the error > >>> > > > >>> > > java.lang.UnsatisfiedLinkError: /tmp/ > >>> > > snappy-1.1.10-b0757287-8557-44b9-9499-afca52f102ec-libsnappyjava.so: > >>> > > Error relocating /lib/ld-linux-x86-64.so.2: unsupported relocation > >>> type 37. > >>> > > > >>> > > Reproduce steps: > >>> > > 1. docker run -it -u root --platform=linux/amd64 > >>> > > czcoder/pulsar:3.3.0-0771f81 bash > >>> > > 2. export > >>> > > PULSAR_EXTRA_OPTS="-Dzookeeper.snapshot.compression.method=snappy" > >>> > > 3. export PULSAR_STANDALONE_USE_ZOOKEEPER=true > >>> > > 4. bin/pulsar standalone -nss -nfw > >>> > > > >>> > > Then you will get the error. More information: > >>> > > https://github.com/sgerrand/alpine-pkg-glibc/issues/194 > >>> > > > >>> > > we need to resolve it before publishing the new release. > >>> > > > >>> > > Yong > >>> > > > >>> > > On Fri, 24 May 2024 at 13:00, guo jiwei > >>> wrote: > >>> > > > >>> > >> +1 (binding) > >>> > >> > >>> > >> - Built from source > >>> > >> - Checked the signatures > >>> > >> - Run standalone > >>> > >> - Checked producer and consumer > >>> > >> - Verified the Cassandra connector > >>> > >> - Verified the Stateful function > >>> > >> > >>> > >> > >>> > >> Regards > >>> > >> Jiwei Guo (Tboy) > >>> > >> > >>> > >> > >>> > >> On Fri, May 24, 2024 at 10:28 AM Cong Zhao > >>> wrote: > >>> > >> > >>> > >> > Hello Apache Pulsar Community, > >>> > >> &g
Re: [VOTE] PIP-353: Improve transaction message visibility for peek-messages
+1 (binding) BR, Zike Yang On Wed, May 22, 2024 at 9:55 PM Baodi Shi wrote: > > Hi, all > > I would like to start the voting thread for PIP-353. > https://github.com/apache/pulsar/pull/22746 > > The implementation PR is: > https://github.com/apache/pulsar/pull/22762 > > Discuss thread: > https://lists.apache.org/thread/dc7f64jtvg987ydztxpffqx88bp44lwv > > Thanks, > Baodi Shi
Re: [DISCUSS] PIP-353: Improve transaction message visibility for peek-messages
LGTM. +1 BR, Zike Yang On Tue, May 21, 2024 at 10:21 PM 太上玄元道君 wrote: > > +1 > > Baodi Shi 于2024年5月20日 周一18:48写道： > > > Hi all, I push a pip to improve transaction message visibility for the > > peek-messages command. > > > > https://github.com/apache/pulsar/pull/22746 > > > > Please feel free to leave your ideas. > > > > > > Thanks, > > Baodi Shi > >
Re: [VOTE] PIP-347: add role field in consumer's stat
+1 (binding) BR, Zike Yang On Tue, May 14, 2024 at 6:56 PM Zixuan Liu wrote: > > +1 (non-binding) > > Thanks, > Zixuan > > PengHui Li 于2024年5月14日周二 18:08写道： > > > +1 (binding) > > > > Regards, > > Penghui > > > > On Tue, May 14, 2024 at 5:50 PM Enrico Olivelli > > wrote: > > > > > +1 (binding) > > > > > > Enrico > > > > > > Il giorno mar 14 mag 2024 alle ore 11:31 太上玄元道君 ha > > > scritto: > > > > > > > +1 nonbinding > > > > > > > > Thanks, > > > > Tao Jiuming > > > > > > > > thetumbled 于2024年5月14日 周二17:26写道： > > > > > > > > > Hi, Pulsar Community. > > > > > I would like to start the voting thread for PIP-347: add role field > > > in > > > > > consumer's stat. > > > > > Proposal PR: https://github.com/apache/pulsar/pull/22564 > > > > > Implementation PR: https://github.com/apache/pulsar/pull/22562 > > > > > > > > > > Thanks, > > > > > Wenzhi Feng(thetumbled). > > > > > > > > >
Re: [VOTE] PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis
+1 (binding) Thanks, Zike Yang On Mon, May 13, 2024 at 3:08 PM ZhangJian He wrote: > > +1(nonbinding) > > Thanks > ZhangJian He > Twitter: shoothzj > Wechat: shoothzj > > > On Mon, May 13, 2024 at 2:35 PM Hang Chen wrote: > > > +1 (binding) > > > > Thanks, > > Hang > > > > 太上玄元道君 于2024年5月13日周一 11:30写道： > > > > > > +1 nonbinding > > > > > > Thanks, > > > Tao Jiuming > > > > > > Yong Zhang 于2024年5月13日周一 10:57写道： > > > > > > > Hi, > > > > > > > > I would like to start voting thread for PIP-350. > > > > https://github.com/apache/pulsar/pull/22688 > > > > > > > > The implementation PR is: > > > > https://github.com/apache/pulsar/pull/22689 > > > > > > > > Discuss thread: > > > > https://lists.apache.org/thread/7tlpkcm2933ddg95kgrb42943r4gq3v9 > > > > > > > > Thanks, > > > > Yong > > > > > >
Re: [VOTE] PIP-348: Trigger offload on topic load stage
+1 (binding) Thanks, Zike Yang On Wed, May 8, 2024 at 6:22 PM Yubiao Feng wrote: > > +1 (binding) > > Thanks > Yubiao Feng > > On Tue, May 7, 2024 at 11:27 AM Hang Chen wrote: > > > Hi guys, > > I want to start voting for PIP-348. > > > > PIP: https://github.com/apache/pulsar/pull/22650 > > PR: https://github.com/apache/pulsar/pull/22652 > > > > DISCUSSION Thread: > > https://lists.apache.org/thread/2ndomp8v4wkcykzthhlyjqfmswor88kv > > > > Thanks, > > Hang > >
Re: [VOTE] Pulsar Node.js Client Release 1.11.0 Candidate 4
+1 (binding) - Verified checksum and signature - Built from source on macOS arm64 and run the example - Installed from the npm registry using node 12,14,16,18,20 on ubuntu - Installed from the npm registry on macos - Ran the example BR, Zike Yang On Tue, Apr 16, 2024 at 8:38 AM 津田秀介 wrote: > > +1 (non-binding) > > - verified checksums and signatures > - confirmed that the build was successful > - ran producer/consumer > > Thanks, > Shusuke Tsuda > > -Original Message- > From: "坂本雅宏" > To: ; > Cc: > Sent: 2024/04/15(月) 17:25 (GMT+09:00) > Subject: Re: [VOTE] Pulsar Node.js Client Release 1.11.0 Candidate 4 > > +1 (binding) > > - verified checksums and signatures > - confirmed that the build was successful > - ran producer/consumer > > Regards, > > > Masahiro Sakamoto > > LY Corporation > Email massa...@lycorp.co.jp > > > > -Original Message- > From: "Baodi Shi" > To: ; > Cc: > Sent: 2024/04/13(土) 18:32 (GMT+09:00) > Subject: Re: [VOTE] Pulsar Node.js Client Release 1.11.0 Candidate 4 > > +1(binding) > > - Verified sign and checksum > - Build from source on macOS arm64 (Node 18) > - Install from npm registry on Windows x64 (Node 18) > - Run ProducerSample and ConsumerSample on macOS (Node 18) > - Installed from the npm registry on Ubuntu using Node v12, v13, v14 ~ v20 > > Thanks, > Baodi Shi > > > On Apr 13, 2024 at 17:30:23, Baodi Shi wrote: > > > Hi everyone, > > > > This is the first release candidate for Apache Pulsar Node.js client, > > version 1.11.0. > > > > It fixes the following issues: > > - https://github.com/apache/pulsar-client-node/milestone/16?closed=1 > > - Related cpp client issue: > > https://github.com/apache/pulsar-client-cpp/compare/v3.4.2...v3.5.1 > > > > Please download the source files and review this release candidate: > > - Download the source package, verify shasum and asc > > - Follow the README.md to build and run the Pulsar Node.js client. > > > > The release candidate package has been published to the npm registry: > > https://www.npmjs.com/package/pulsar-client/v/1.11.0-rc.4 > > You can install it by `npm i pulsar-client@1.11.0-rc.4 > > --pulsar_binary_host_mirror= > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/` and > > verify the package. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source files: > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.11.0-rc.4/ > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://downloads.apache.org/pulsar/KEYS > > > > SHA-512 checksum: > > 52cd6949e08d773ac11d315765278feb0165b65cabe654fb5d270acf17dffa77442ca4d42ad40b5ff7be92140a84ec98c1c94c616db84aff416641e43599bf5b > > ./apache-pulsar-client-node-1.11.0.tar.gz > > > > The tag to be voted upon: > > v1.11.0-rc.4 > > https://github.com/apache/pulsar-client-node/releases/tag/v1.11.0-rc.4 > > > > Please review and vote on the release candidate #1 for the version 1.11.0, > > as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > Thanks, > > Baodi Shi > > >
Re: [VOTE] Pulsar Client Python Release 3.5.0 Candidate 3
+1 (binding) - Verified checksums and signatures - Installed Python client using the wheel file on macOS using Python 3.9/3.10 and ran the examples - Built from source on macOS and ran the example BR, Zike Yang On Thu, Apr 11, 2024 at 6:40 PM Yunze Xu wrote: > > +1 (binding) > > - Verified signatures and checksums > - Built from source from source on macOS (with C++ client 3.5.1) > - Verified e2e with OAuth2 authentication for the following wheels: > - Verified e2e with > pulsar_client-3.5.0-cp312-cp312-macosx_10_15_universal2.whl > > Thanks, > Yunze > > On Sun, Apr 7, 2024 at 11:16 PM guo jiwei wrote: > > > > +1 (binding) > > > > - Checked the signature and checksums > > - Install the pulsar_client-3.5.0-cp310-cp310-macosx_10_15_universal2.whl > > - Run Pulsar standalone > > - Run the Python examples producer and consumer > > > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Tue, Apr 2, 2024 at 5:49 PM Yunze Xu wrote: > > > > > This is the 3rd release candidate for Apache Pulsar Client Python, > > > version 3.5.0. > > > > > > It fixes the following issues: > > > https://github.com/apache/pulsar-client-python/milestone/6?closed=1 > > > > > > *** Please download, test and vote on this release. This vote will > > > stay open for at least 72 hours *** > > > > > > Python wheels: > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.5.0-candidate-3/ > > > > > > The supported python versions are 3.8, 3.9, 3.10, 3.11 and 3.12. The > > > supported platforms and architectures are: > > > - Windows x86_64 (windows/) > > > - glibc-based Linux x86_64 (linux-glibc-x86_64/) > > > - glibc-based Linux arm64 (linux-glibc-arm64/) > > > - musl-based Linux x86_64 (linux-musl-x86_64/) > > > - musl-based Linux arm64 (linux-musl-arm64/) > > > - macOS universal 2 (macos/) > > > > > > You can download the wheel (the `.whl` file) according to your own OS > > > and Python version > > > and install the wheel: > > > - Windows: `py -m pip install *.whl --force-reinstall` > > > - Linux or macOS: `python3 -m pip install *.whl --force-reinstall` > > > > > > The tag to be voted upon: v3.5.0-candidate-3 > > > (38737a24f6ba77fe4efc5321980513ae317920e4) > > > > > > https://github.com/apache/pulsar-client-python/releases/tag/v3.5.0-candidate-3 > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > > https://downloads.apache.org/pulsar/KEYS > > > > > > Please download the Python wheels and follow the README to test. > > >
Re: [VOTE] Pulsar Node.js Client Release 1.11.0 Candidate 3
- Verified checksum and signature
- Built from source on macOS arm64 and run the example
- Installed from the npm registry and run the example
- Installed from the npm registry on Ubuntu using Node v14, v16, v18
and run the example
Installed from the npm registry using node v10, v11, v12, v13 but if
failed with the following errors:
```
> pulsar-client@1.11.0-rc.3 install
> /data/verify/node1-11/node_modules/pulsar-client
> node-pre-gyp install --fallback-to-build && node GenCertFile.js
[pulsar-client] Success:
"/data/verify/node1-11/node_modules/pulsar-client/lib/binding/pulsar.node"
is installed via remote
/data/verify/node1-11/node_modules/pulsar-client/src/Client.js:60
fs.rmSync(certsFilePath, { force: true });
^
TypeError: fs.rmSync is not a function
at Function.genCertFile
(/data/verify/node1-11/node_modules/pulsar-client/src/Client.js:60:8)
at Object.
(/data/verify/node1-11/node_modules/pulsar-client/GenCertFile.js:22:8)
at Module._compile (internal/modules/cjs/loader.js:1118:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1138:10)
at Module.load (internal/modules/cjs/loader.js:982:32)
at Function.Module._load (internal/modules/cjs/loader.js:875:14)
at Function.executeUserEntryPoint [as runMain]
(internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47
npm WARN node1-11@1.0.0 No description
npm WARN node1-11@1.0.0 No repository field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! pulsar-client@1.11.0-rc.3 install: `node-pre-gyp install
--fallback-to-build && node GenCertFile.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the pulsar-client@1.11.0-rc.3 install script.
npm ERR! This is probably not a problem with npm. There is likely
additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/zike/.npm/_logs/2024-04-12T02_40_33_159Z-debug.log
```
However the README states that the client is compatible with Nodejs 10
or later:
https://github.com/apache/pulsar-client-node/blob/e3bf582ea450e68379ff685598257ed6e69fb1aa/README.md?plain=1#L26
Perhaps we should update our version compatibility strategy in the
README.
BR,
Zike Yang
On Fri, Apr 12, 2024 at 8:13 AM Baodi Shi wrote:
>
> +1(binding)
>
> - verified sign and checksum
> - Build from source and install from npm registry on macOS arm64
> - Install from npm registry on Windows x64
> - Run ProducerSample and ConsumerSample on macOS
>
>
> Thanks,
> Baodi Shi
>
>
> On Apr 11, 2024 at 18:43:51, Yunze Xu wrote:
>
> > +1 (binding)
> >
> > - Verified signature and checksums
> > - Built from source on macOS
> > - Run e2e tests against a local standalone
> > - Run e2e tests with OAuth2 authentication against an online cluster
> >
> > Thanks,
> > Yunze
> >
> > On Tue, Apr 9, 2024 at 6:26 PM Baodi Shi wrote:
> >
> >
> > Hi everyone,
> >
> >
> > This is the first release candidate for Apache Pulsar Node.js client,
> >
> > version 1.11.0.
> >
> >
> > It fixes the following issues:
> >
> > - https://github.com/apache/pulsar-client-node/milestone/16?closed=1
> >
> > - Related cpp client issue:
> >
> > https://github.com/apache/pulsar-client-cpp/compare/v3.4.2...v3.5.1
> >
> >
> > Please download the source files and review this release candidate:
> >
> > - Download the source package, verify shasum and asc
> >
> > - Follow the README.md to build and run the Pulsar Node.js client.
> >
> >
> > The release candidate package has been published to the npm registry:
> >
> > https://www.npmjs.com/package/pulsar-client/v/1.11.0-rc.3
> >
> > You can install it by `npm i pulsar-client@1.11.0-rc.3
> >
> > --pulsar_binary_host_mirror=
> >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/` and
> >
> > verify the package.
> >
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> >
> > approval, with at least 3 PMC affirmative votes.
> >
> >
> > Source files:
> >
> >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.11.0-rc.3/
> >
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> >
> > https://downloads.apache.org/pulsar/KEYS
> >
> >
> > SHA-512 checksum:
> >
> >
> > 74196c05f80a0c2569c1d19d7e21ea18babb5fe75fc5936c6eef47574ee7a9abc0f4a7e4db3b740de760d477cb2dfc51c9fd5b2dd343ee54389d78b1e24849e7
> >
> > ./apache-pulsar-client-node-1.11.0.tar.gz
> >
> >
> > The tag to be voted upon:
> >
> > v1.11.0-rc.3
> >
> > https://github.com/apache/pulsar-client-node/releases/tag/v1.11.0-rc.3
> >
> >
> > Please review and vote on the release candidate #1 for the version 1.11.0,
> >
> > as follows:
> >
> > [ ] +1, Approve the release
> >
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> >
> > Thanks,
> >
> > Baodi Shi
> >
> >
Re: [DISCUSS] Plan for Pulsar Go Client 1.0.0
> BTW, there is another issue that needs to be considered. > https://github.com/apache/pulsar-client-go/issues/919 Thanks. Yes, we need to consider that. BR, Zike Yang On Tue, Apr 2, 2024 at 10:50 AM Zixuan Liu wrote: > > > We maintain two versions of the API, but they could share the same > internal code. > > Thanks for your explanation! +1 > > > Jie crossover 于2024年4月1日周一 22:20写道： > > > Greet plan. > > > We maintain two versions of the API, but they could share the same > > > internal code. I don't think this will add much complexity. > > > > I agree with this. > > BTW, there is another issue that needs to be considered. > > > > https://github.com/apache/pulsar-client-go/issues/919 > > > > > > -- > > Best Regards！ > > crossoverJie > > > > > > Yunze Xu 于2024年4月1日周一 20:46写道： > > > > > > This would lead to inconsistency in API definitions, making them > > > > appear disorganized. > > > > > > I agree. Using such suffixes is really ugly and much harder to > > > maintain IMHO. It only makes people add APIs more casually. They might > > > think, oh, don't worry, if this API (e.g. doSomething()) does not make > > > sense in future, you can add another API with a suffix (e.g. > > > doSomethingInAnotherWay()). > > > > > > Thanks, > > > Yunze > > > > > > On Mon, Apr 1, 2024 at 8:05 PM Zike Yang wrote: > > > > > > > > Thanks for all your comments. I address each one in line. > > > > > > > > > Regarding the `Close` method, I think it's exceptional. It's designed > > > > to return no error so that users can call `defer client.Close()` [1]. > > > > To process errors in `Close()`, we can panic in the implementation and > > > > let users use `recover` to catch the panic. > > > > > > > > My initial idea was to implement the `io.Closer` interface to allow > > > > users to free up resources in a general way. After investigating this > > > > discussion: https://github.com/golang/go/issues/40483, I think that > > > > the close method for Pulsar resources should not return an error. The > > > > client should ensure all resources are released when closed, > > > > regardless of whether any error has occurred. So this makes sense to > > > > me. > > > > > > > > > I’d add one more item for consideration: using WithX() methods > > > instead of > > > > a strict for configurations: > > > > https://github.com/apache/pulsar-client-go/issues/68 > > > > > > > > Thanks for sharing this issue. It makes sense to me. > > > > > > > > > We can add a method with context, so like: CloseWithContext. > > > > > We can still follow the above way, so like: CloseWithError. > > > > > > > > This would lead to inconsistency in API definitions, making them > > > > appear disorganized. For instance, some methods accept a context but > > > > don't end with "WithContext.". Same for the `WithError`. This also > > > > adds complexity to the API, resulting in a poor user experience. > > > > Moreover, we should actually deprecate the old APIs, which would make > > > > names like CloseWithCtx and FlushWithCtx seem odd. > > > > > > > > > In the ConsumerOptions, we use the `internal` package, this is > > > incorrect, > > > > when you set the value for `BackoffPolicy`, you will see this error > > `Use > > > of > > > > the internal package is not allowed`. > > > > For internal package, please see > > > https://go.dev/doc/go1.4#internalpackages > > > > > > > > Yes. This issue https://github.com/apache/pulsar-client-go/issues/1187 > > > > is actually tracking on it. > > > > > > > > > This will increase our maintenance times. > > > > > > > > We maintain two versions of the API, but they could share the same > > > > internal code. I don't think this will add much complexity. On the > > > > contrary, it would make our API maintenance clearer and easier. The > > > > approach mentioned above, using WithContext and WithError, is actually > > > > also increasing the maintenance workload. I believe the complexity > > > > and workload it brings is not less than maintaining two versions, v1 > > > > and v0, of our API. > > > > > > > > BR, &
Re: [DISCUSS] Plan for Pulsar Go Client 1.0.0
Thanks for all your comments. I address each one in line. > Regarding the `Close` method, I think it's exceptional. It's designed to return no error so that users can call `defer client.Close()` [1]. To process errors in `Close()`, we can panic in the implementation and let users use `recover` to catch the panic. My initial idea was to implement the `io.Closer` interface to allow users to free up resources in a general way. After investigating this discussion: https://github.com/golang/go/issues/40483, I think that the close method for Pulsar resources should not return an error. The client should ensure all resources are released when closed, regardless of whether any error has occurred. So this makes sense to me. > I’d add one more item for consideration: using WithX() methods instead of a strict for configurations: https://github.com/apache/pulsar-client-go/issues/68 Thanks for sharing this issue. It makes sense to me. > We can add a method with context, so like: CloseWithContext. > We can still follow the above way, so like: CloseWithError. This would lead to inconsistency in API definitions, making them appear disorganized. For instance, some methods accept a context but don't end with "WithContext.". Same for the `WithError`. This also adds complexity to the API, resulting in a poor user experience. Moreover, we should actually deprecate the old APIs, which would make names like CloseWithCtx and FlushWithCtx seem odd. > In the ConsumerOptions, we use the `internal` package, this is incorrect, when you set the value for `BackoffPolicy`, you will see this error `Use of the internal package is not allowed`. For internal package, please see https://go.dev/doc/go1.4#internalpackages Yes. This issue https://github.com/apache/pulsar-client-go/issues/1187 is actually tracking on it. > This will increase our maintenance times. We maintain two versions of the API, but they could share the same internal code. I don't think this will add much complexity. On the contrary, it would make our API maintenance clearer and easier. The approach mentioned above, using WithContext and WithError, is actually also increasing the maintenance workload. I believe the complexity and workload it brings is not less than maintaining two versions, v1 and v0, of our API. BR, Zike Yang On Sat, Mar 30, 2024 at 7:34 PM Zixuan Liu wrote: > > I don't recommend breaking user APIs. There are many ways we can avoid it. > > > 1. We should support passing the context to some IO-related methods > like `Ack`, `Close`, `Flush`, etc, or any other methods. There is an > issue related to this discussion. [2] > > We can add a method with context, so like: CloseWithContext. > > > 2. Some methods need to return the error. Like `Reader.HasNext`, > `Close`. This is to adhere to Golang's error-handling standards. > > We can still follow the above way, so like: CloseWithError. > > > 4. Some APIs need to be refined and require introducing breaking > changes as they could affect user experience otherwise. For example, > this [4] is an issue discussing the redesign of the Backoff Policy > API. > > In the ConsumerOptions, we use the `internal` package, this is incorrect, > when you set the value for `BackoffPolicy`, you will see this error `Use of > the internal package is not allowed`. > For internal package, please see https://go.dev/doc/go1.4#internalpackages > > For incorrect implementation, we can fix this. > > > We can provide a separate package path for v1.x API versions, > maintaining v0.x and v1.x APIs separately. > > This will increase our maintenance times. > > > Matteo Merli 于2024年3月30日周六 13:08写道： > > > The plan looks great. > > > > I’d add one more item for consideration: using WithX() methods instead of > > a strict for configurations: > > https://github.com/apache/pulsar-client-go/issues/68 > > > > > > -- > > Matteo Merli > > > > > > > > On Fri, Mar 29, 2024 at 5:38 AM Zike Yang wrote: > > > > > Hi, everyone > > > > > > The Pulsar Go Client[0] is now relatively mature. I've also noticed > > > that many people in the community have widely used it in their > > > production environments. However, there's still no official 1.0 > > > version for the Go client. I would like to start a thread to discuss > > > the plan for releasing Go client 1.0.0. > > > > > > According to "Go Module version numbering" [1], there are strict > > > requirements for version management in Golang projects, which means we > > > can't introduce any breaking changes within a major version. Before > > > releasing version 1.0.0, we need to review our API and decide on the > > > finalized API for Go client
[DISCUSS] Plan for Pulsar Go Client 1.0.0
Hi, everyone The Pulsar Go Client[0] is now relatively mature. I've also noticed that many people in the community have widely used it in their production environments. However, there's still no official 1.0 version for the Go client. I would like to start a thread to discuss the plan for releasing Go client 1.0.0. According to "Go Module version numbering" [1], there are strict requirements for version management in Golang projects, which means we can't introduce any breaking changes within a major version. Before releasing version 1.0.0, we need to review our API and decide on the finalized API for Go client 1.0.0. I've observed that the current design of the Go client's API still has the following issues: 1. We should support passing the context to some IO-related methods like `Ack`, `Close`, `Flush`, etc, or any other methods. There is an issue related to this discussion. [2] 2. Some methods need to return the error. Like `Reader.HasNext`, `Close`. This is to adhere to Golang's error-handling standards. 3. We should expose errors to users so that users can inspect the types of errors returned. [3] is an issue to discuss about this. 4. Some APIs need to be refined and require introducing breaking changes as they could affect user experience otherwise. For example, this [4] is an issue discussing the redesign of the Backoff Policy API. Additionally, we need to continue standardizing the release process and fixing known issues: 1. Refine the changelog formt [5]. We could try to utilize the tool "go-changelog" [6] to generate the changelog automatically. 2. Refine the release process [7] to adhere the Golang Moduel version standard. We need to clearly define the compatibility relationships between different types of versions. Some processes may need to be adjusted to comply with these version standards. These API changes will inevitably introduce breaking changes. However, we do not want the release of Go client 1.0.0 to cause troublesome impacts on the upgrade process for all our existing users. Inspired by the blog "The Principles of Versioning in Go" [8], I believe we need to follow this principle in the process of releasing 1.0.0 and also for maintaining subsequent versions: We should strive to avoid introducing breaking changes to the existing APIs and behaviors. We aim to reduce the steps needed for users to upgrade to the major version. To achieve this, I would like to suggest the following basic solution: We can provide a separate package path for v1.x API versions, maintaining v0.x and v1.x APIs separately. At the same time, we will deprecate all v0.x APIs. For future major versions like 2.x, 3.x, we will follow this same approach according to Golang's standards. In this way, when users upgrade to 1.0.0, they can gradually modify their code to utilize the new version of API, while still being able to use the features of the old API. We will remove the v0.x API in a later version, perhaps in version 2.0.0. The structure for the v1 package would look like this: ├── go.mod# Note: The v0 APIs and v1 APIs will shared the same go.mod ├── v1 │ └── pulsar │ └── client.go └── pulsar └── client.go I did a small demo. You can check it here: https://github.com/RobertIndie/test-go [9]. In this way, the user could still use `go get github.com/apache/pulsar-client-go@v1.0.0` to upgrade to the v1.0.0 version. And use `import "github.com/apache/pulsar-client-go/v1/pulsar"` to use the new V1 API. And for the future major versions like v2.0.0. The users could use `go get github.com/apache/pulsar-client-go/v2` to install the client and use `import "github.com/apache/pulsar-client-go/v2/pulsar` to use the V2 API. While Golang's versioning standard allows us to introduce breaking changes to the v0.x API, I favor preserving the current API. Considering the resistance our users have towards upgrades, I am more inclined to avoid breaking changes to the existing API. This approach should reduce the impact of upgrades. For our next steps, we could proceed as follows: 1. Discuss and finalize the Go client version strategy to adhere to the "Golang Module version standard"[1] 2. Discuss and finalize the V1 API. We may need a PIP to finalize it. 3. Add the V1 API, develop the necessary features, and address issues based on this new API. 4. Release the Go Client 1.0.0 based on the refined release process. This is currently just a preliminary idea and plan for the release of Go client 1.0. I would like to hear your thoughts and ideas. What are your thoughts on this proposal? What else do you believe we need to do before releasing Go client v1.0.0? Please feel free to add any points I may have missed, and feel free to leave your comments here. Thanks, Zike Yang [0] https://github.com/apache/pulsar-client-go [1] https://go.dev/doc/modules/version-numbers [2] https://github.com/apache/pulsar-client-go/issues/1170 [3] https://github.com/apa
Re: [VOTE] PIP-344: Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName)
+1 (no-binding) Thanks, Zike Yang On Thu, Mar 21, 2024 at 11:42 AM Yunze Xu wrote: > > +1 (binding) > > Thanks, > Yunze > > On Wed, Mar 20, 2024 at 9:19 PM PengHui Li wrote: > > > > +1 (binding) > > > > Regards. > > Penghui > > > > On Sat, Mar 16, 2024 at 6:28 AM Yubiao Feng > > wrote: > > > > > Hi All > > > > > > This thread is to start a vote for PIP-344. > > > > > > PIP: https://github.com/apache/pulsar/pull/22182 > > > Discussion thread: > > > https://lists.apache.org/thread/z693blcxoqk0mj0rzyt1k7nvy72j18t5 > > > > > > Thanks > > > Yubiao Feng > > >
Re: [DISCUSS] Release Pulsar Python Client 3.5.0
+1 Thanks, Zike Yang On Tue, Mar 19, 2024 at 8:00 PM Yunze Xu wrote: > > Hi all, > > I would like to propose releasing the Pulsar Python client 3.5.0. > > It's over two months since the release of 3.4.0, and there are 16 new > commits: https://github.com/apache/pulsar-client-python/compare/v3.4.0...main. > The dependent C++ client was upgraded to 3.5.0 in > https://github.com/apache/pulsar-client-python/pull/202, which also > fixes some bugs. It's time to cut a new release. > > Please let me know if you have any important fixes that need to be > included in Pulsar Python client 3.5.0. > > Thanks, > Yunze
Re: [DISCUSS] Pulsar Client Go - Avro Schema
Hi,
> A) Should the Consumer / Reader be allowed to override the Schema creation
after schema-definition is retrieved from Schema Registry ?
Are you suggesting we adjust the schema creator, allowing the
Consumer/Reader to use a different Avro library? I'm interested in
others' thoughts.
> B) Should the Consumer / Reader have access to query Schema Registry for
retrieving the schema-definition of the message ?
This has already been implemented. The Consumer/Reader attempts to
fetch the writer's schema from the registry to decode the message. If
it fails, it uses the consumer reader's schema for decoding.
> C) Should just drop linkedin/goavro and migrate to
https://github.com/hamba/avro because: avro to golang struct generation ->
see avrogen; Easier handling of "nullable" union. by having a field as a
pointer; benchmark shows to be faster than linkedin.
Thanks for bringing the hamba/avro to the discussion.
Regarding the avro to golang struct generation, we can use
`gogen-avro` for the goavro.
Considering the 'nullable' union and benchmark results, hamba/avro
appears to be more user-friendly and faster.
Below, I've provided two examples to illustrate the differences in how
these two Avro libraries handle nullable unions.
type TestHambaAvro struct {
Age *int `avro:"age"`
}
type TestGoavro struct {
Age map[string]interface{} `avro:"age"`
}
Clearly, handling the 'age' field with hamba/avro is more intuitive.
In contrast, using goavro requires us to define a
map[string]interface{} type.
I'd like to hear from others. Would it be reasonable to provide an
option for go client users to switch between these two avro libraries?
Thanks,
Zike Yang
On Fri, Mar 8, 2024 at 4:33 AM Adrian Iacob-Ghiula
wrote:
>
> Hi Everybody,
>
> Pulsar Client Go uses https://github.com/linkedin/goavro for avro encoded
> messages. pulsar.Schema is automatically created with no way to override
> when a message has a schema version.
> A Consumer / Reader has access to the raw bytes of the message but does not
> have access to the schema-definition of the message.
>
> A) Should the Consumer / Reader be allowed to override the Schema creation
> after schema-definition is retrieved from Schema Registry ?
> B) Should the Consumer / Reader have access to query Schema Registry for
> retrieving the schema-definition of the message ?
> C) Should just drop linkedin/goavro and migrate to
> https://github.com/hamba/avro because: avro to golang struct generation ->
> see avrogen; Easier handling of "nullable" union. by having a field as a
> pointer; benchmark shows to be faster than linkedin.
>
> I would go for the flexibility as it will not break consumers using
> linkedin/goavro but advice would be nice :)
>
> Thanks
[ANNOUNCE] Apache Pulsar Go Client 0.12.1 released
The Apache Pulsar team is proud to announce Apache Pulsar Go Client version 0.12.1. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar release details and downloads, visit: https://github.com/apache/pulsar-client-go/releases/tag/v0.12.1 Release Notes are at: https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG.md We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
Re: [VOTE] Pulsar Client Go Release 0.12.1 Candidate 1
Thanks to everyone! Close this vote with 3 bindings +1s: - Yunze - Matteo Merli - Penghui Thanks, Zike Yang On Fri, Mar 8, 2024 at 1:08 PM PengHui Li wrote: > > +1 (binding) > > - Checked the signatures > - Build and Run the local test. > > Regards, > Penghui > > On Fri, Mar 8, 2024 at 12:55 PM Matteo Merli wrote: > > > +1 > > > > Verified checksums and signatures > > Verified tests and local perf client > > > > -- > > Matteo Merli > > > > > > > > On Wed, Mar 6, 2024 at 9:23 PM Yunze Xu wrote: > > > > > +1 (binding) > > > > > > - Verified checksum and signatures > > > - Run OAuth2 e2e examples with 0.12.1-candidate-1 > > > - Built from source and run perf to produce and consume > > > > > > Thanks, > > > Yunze > > > > > > On Thu, Feb 29, 2024 at 11:05 AM Zike Yang wrote: > > > > > > > > Hi everyone, > > > > Please review and vote on the release candidate #1 for the version > > > > 0.12.1, as follows: > > > > [ ] +1, Approve the release > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > This is the first release candidate for Apache Pulsar Go client, > > version > > > 0.12.1. > > > > > > > > The release note/changelog for Go client 0.12.1: > > > > https://github.com/apache/pulsar-client-go/pull/1189/files > > > > > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this > > > release: > > > > https://downloads.apache.org/pulsar/KEYS > > > > > > > > Please download these packages and review this release candidate: > > > > - Review release notes: > > > https://github.com/apache/pulsar-client-go/pull/1189 > > > > - Download the source package (verify shasum, and asc) and follow the > > > > README.md to build and run the pulsar-client-go. > > > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > Source file: > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.1-candidate-1/ > > > > > > > > The tag to be voted upon: > > > > v0.12.1-candidate-1 > > > > https://github.com/apache/pulsar-client-go/tree/v0.12.1-candidate-1 > > > > > > > > SHA-512 checksums: > > > > > > > > > 691a301d099a602baa30bb7ec22dc33b8503d5ad5aa5fe43f51aab188099e4781844070de11584fd82d40227a28fe763ed8bd2cfa93b99d73ebfd34c6061e02d > > > > apache-pulsar-client-go-0.12.1-src.tar.gz > > > > >
[VOTE] Pulsar Client Go Release 0.12.1 Candidate 1
Hi everyone, Please review and vote on the release candidate #1 for the version 0.12.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the first release candidate for Apache Pulsar Go client, version 0.12.1. The release note/changelog for Go client 0.12.1: https://github.com/apache/pulsar-client-go/pull/1189/files Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://downloads.apache.org/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1189 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.1-candidate-1/ The tag to be voted upon: v0.12.1-candidate-1 https://github.com/apache/pulsar-client-go/tree/v0.12.1-candidate-1 SHA-512 checksums: 691a301d099a602baa30bb7ec22dc33b8503d5ad5aa5fe43f51aab188099e4781844070de11584fd82d40227a28fe763ed8bd2cfa93b99d73ebfd34c6061e02d apache-pulsar-client-go-0.12.1-src.tar.gz
Re: [VOTE] PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources
+1 (no-binding) BR, Zike Yang On Tue, Feb 27, 2024 at 8:56 AM PengHui Li wrote: > > +1 (binding) > > Regards, > Penghui > > On Mon, Feb 26, 2024 at 5:44 PM Pengcheng Jiang > wrote: > > > Hi, community > > > > I'm starting the vote for PIP-339: Introducing the --log-topic Option for > > Pulsar Sinks and Sources > > PIP link: https://github.com/apache/pulsar/pull/22071 > > > > Thanks, > > Pengcheng Jiang > >
Re: [ANNOUNCE] New Committer: Asaf Mesika
Congratulations! BR, Zike Yang On Wed, Feb 21, 2024 at 2:22 PM Hang Chen wrote: > > Congratulations Asaf! > > Best, > Hang > > 太上玄元道君 于2024年2月21日周三 13:07写道： > > > > congrats! > > > > Lari Hotari 于2024年2月21日 周三00:50写道： > > > > > The Apache Pulsar Project Management Committee (PMC) has invited > > > Asaf Mesika https://github.com/asafm to become a committer and we > > > are pleased to announce that he has accepted. > > > > > > Welcome and Congratulations, Asaf Mesika! > > > > > > Please join us in congratulating and welcoming Asaf onboard! > > > > > > Best Regards, > > > > > > Lari Hotari > > > on behalf of the Pulsar PMC > > >
Re: [DISCUSS] PIP-331: WASM Function API
Thanks for proposing this feature! I replied in the PR. Before reviewing this PIP, I thought it was an implementation of a WASM function runtime. IIUC, for this PIP, users need to first define a Java Pulsar function and then construct the FunctionMap to interact with the wasm. Right? Just share my idea here: What do you think about implementing a WASM function runtime directly? It could be on par with the current Java function runtime. If we define the API or SDK for WASM and Pulsar interaction, users could simply upload their WASM files and process messages just like a Pulsar function. BR, Zike Yang On Sun, Feb 18, 2024 at 10:12 PM Asaf Mesika wrote: > > Hi ZiCheng, > > Brilliant suggestion! > > I replied in the PR section, which I couldn't understand. > > > On Tue, Jan 30, 2024 at 1:18 PM dragon-zhang > wrote: > > > Hi Pulsar Community, > > > > I want to add a new feature that supports run WASM bytecode to the > > pulsar-functions module. > > > > Please see the PIP: https://github.com/apache/pulsar/pull/21992 > > > > Thanks, > > ZiCheng Zhang
Re: [Discuss] Release Pulsar C++ Client 3.5.0
+1 Thanks for driving the release. BR, Zike Yang On Wed, Feb 7, 2024 at 2:20 PM Yunze Xu wrote: > > I would like to propose releasing the Pulsar C++ Client 3.5.0. > > It has been about 2 months since the last release (3.4.2). There have > been many new features and bug fixes since then. It's time to release > a new version. I will start the release once > https://github.com/apache/pulsar-client-cpp/pull/401 is ready. > > Please let me know if you have any PRs that need to be included in 3.5.0. > > Thanks, > Yunze
Re: [VOTE] PIP-335: Oxia metadata support
+1 Thanks, Zike Yang On Tue, Feb 6, 2024 at 6:58 PM Enrico Olivelli wrote: > > +1 (binding) > > Enrico > > Il giorno mar 6 feb 2024 alle ore 11:36 houxiaoyu > ha scritto: > > > > +1 nonbinding > > > > Thanks, > > Xiaoyu Hou > > > > 太上玄元道君 于2024年2月6日周二 15:59写道： > > > > > > +1 nonbinding > > > > > > Matteo Merli 于2024年2月6日 周二04:40写道： > > > > > > > https://github.com/apache/pulsar/pull/22009 > > > > > > > > - > > > > > > > > # PIP-335: Support Oxia metadata store plugin > > > > > > > > # Motivation > > > > > > > > Oxia is a scalable metadata store and coordination system that can be > > > > used > > > > as the core infrastructure > > > > to build large scale distributed systems. > > > > > > > > Oxia was created with the primary goal of providing an alternative > > > > Pulsar > > > > to replace ZooKeeper as the > > > > long term preferred metadata store, overcoming all the current > > > > limitations > > > > in terms of metadata > > > > access throughput and data set size. > > > > > > > > # Goals > > > > > > > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK. > > > > > > > > Users will be able to start a Pulsar cluster using just Oxia, without > > > > any > > > > ZooKeeper involved. > > > > > > > > ## Not in Scope > > > > > > > > It's not in the scope of this proposal to change any default behavior or > > > > configuration of Pulsar. > > > > > > > > # Detailed Design > > > > > > > > ## Design & Implementation Details > > > > > > > > Oxia semantics and client SDK were already designed with Pulsar and > > > > MetadataStore plugin API in mind, so > > > > there is not much integration work that needs to be done here. > > > > > > > > Just few notes: > > > > 1. Oxia client already provides support for transparent batching of > > > > read > > > > and write operations, > > > > so there will be no use of the batching logic in > > > > `AbstractBatchedMetadataStore` > > > > 2. Oxia does not treat keys as a walkable file-system like interface, > > > > with > > > > directories and files. Instead > > > > all the keys are independent. Though Oxia sorting of keys is aware > > > > of > > > > '/' and provides efficient key > > > > range scanning operations to identify the first level children of a > > > > given key > > > > 3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to > > > > exist. eg: we can create `/a/b/c` key > > > > without `/a/b` and `/a` existing. > > > > In the Pulsar integration for Oxia we're forcing to create all > > > > parent > > > > keys when they are not there. This > > > > is due to several places in BookKeeper access where it does not > > > > create > > > > the parent keys, though it will > > > > later make `getChildren()` operations on the parents. > > > > > > > > ## Other notes > > > > > > > > Unlike in the ZooKeeper implementation, the notification of events is > > > > guaranteed in Oxia, because the Oxia > > > > client SDK will use the transaction offset after server reconnections > > > > and > > > > session restarted events. This > > > > will ensure that brokers cache will always be properly invalidated. We > > > > will > > > > then be able to remove the > > > > current 5minutes automatic cache refresh which is in place to prevent > > > > the > > > > ZooKeeper missed watch issue. > > > > > > > > # Links > > > > > > > > Oxia: https://github.com/streamnative/oxia > > > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java > > > > > > > > > > > > -- > > > > Matteo Merli > > > > > > > >
[DISCUSS] Release Pulsar Go Client 0.12.1
We have found 2 regression bugs introduced in 0.12.0 1. SIGSEGV error in 0.12.0 with zstd compression enabled, when the producer is shared between multiple goroutines(https://github.com/apache/pulsar-client-go/issues/1163). This issue has been fixed by https://github.com/apache/pulsar-client-go/pull/1164. Thank @ben(https://github.com/0x4500) for discovering and reporting this issue. 2. Race condition issue when sending multiple messages concurrently. This has also been fixed by https://github.com/apache/pulsar-client-go/pull/1164 These two regression bugs arose because we refactored the producer's publishing logic in version 0.12.0. We moved the message compression and sequence ID assignment from the producer's internal goroutine to the user's goroutine, which led to concurrency problems. We need to trigger a new release 0.12.1 to address these bugs. Please let me know if you have any PRs that need to be included in 0.12.1. BR, Zike Yang
Re: [DISCUSS] PIP-335: Oxia metadata support
+1 Thanks, Zike Yang On Thu, Feb 1, 2024 at 9:10 AM Hang Chen wrote: > > +1 > > Best, > Hang > > PengHui Li 于2024年2月1日周四 09:03写道： > > > > +1 > > > > Best, > > Penghui > > > > On Thu, Feb 1, 2024 at 7:58 AM Matteo Merli wrote: > > > > > https://github.com/apache/pulsar/pull/22009 > > > > > > === > > > > > > # PIP-335: Supporty Oxia metadata store plugin > > > > > > # Motivation > > > > > > Oxia is a scalable metadata store and coordination system that can be used > > > as the core infrastructure > > > to build large scale distributed systems. > > > > > > Oxia was created with the primary goal of providing an alternative Pulsar > > > to replace ZooKeeper as the > > > long term preferred metadata store, overcoming all the current limitations > > > in terms of metadata > > > access throughput and data set size. > > > > > > # Goals > > > > > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK. > > > > > > Users will be able to start a Pulsar cluster using just Oxia, without any > > > ZooKeeper involved. > > > > > > ## Not in Scope > > > > > > It's not in the scope of this proposal to change any default behavior or > > > configuration of Pulsar. > > > > > > # Detailed Design > > > > > > ## Design & Implementation Details > > > > > > Oxia semantics and client SDK were already designed with Pulsar and > > > MetadataStore plugin API in mind, so > > > there is not much integration work that needs to be done here. > > > > > > Just a few notes: > > > 1. Oxia client already provides support for transparent batching of read > > > and write operations, > > > so there will be no use of the batching logic in > > > `AbstractBatchedMetadataStore` > > > 2. Oxia does not treat keys as a walkable file-system like interface, > > > with > > > directories and files. Instead > > > all the keys are independent. Though Oxia sorting of keys is aware of > > > '/' and provides efficient key > > > range scanning operations to identify the first level children of a > > > given key > > > 3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to > > > exist. eg: we can create `/a/b/c` key > > > without `/a/b` and `/a` existing. > > > In the Pulsar integration for Oxia we're forcing to create all parent > > > keys when they are not there. This > > > is due to several places in BookKeeper access where it does not create > > > the parent keys, though it will > > > later make `getChildren()` operations on the parents. > > > > > > ## Other notes > > > > > > Unlike in the ZooKeeper implementation, the notification of events is > > > guaranteed in Oxia, because the Oxia > > > client SDK will use the transaction offset after server reconnections and > > > session restarted events. This > > > will ensure that brokers cache will always be properly invalidated. We > > > will > > > then be able to remove the > > > current 5minutes automatic cache refresh which is in place to prevent the > > > ZooKeeper missed watch issue. > > > > > > # Links > > > > > > Oxia: https://github.com/streamnative/oxia > > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java > > > > > > > > > -- > > > Matteo Merli > > > > > >
Re: [VERIFY] Pulsar Release 3.2.0 Candidate 5
Based on the release policy(https://pulsar.apache.org/contribute/release-policy/), for each feature release, there would be three weeks to verify the release candidate before the release deadline. And the first two weeks are only for verification. I think we need to state the release deadline before each feature release. What’s the deadline of 3.2.0? @Jiwei Maybe we could start the vote now? > We don't have that in our release process documented at > https://pulsar.apache.org/contribute/release-process/ . Indeed, we need to add this guidance into the release process to align it with the release policy. Thanks, Zike Yang On Mon, Jan 29, 2024 at 11:25 PM Lari Hotari wrote: > > Is this a release VOTE thread? I think that the subject of the thread should > state that explicitly. > > What does a VERIFY thread mean? We don't have that in our release process > documented at https://pulsar.apache.org/contribute/release-process/ . > > Thanks, > > -Lari > > On 2024/01/27 15:14:27 guo jiwei wrote: > > This is the fifth release candidate for Apache Pulsar version 3.2.0. > > > > It fixes the following issues: > > https://github.com/apache/pulsar/milestone/36?closed=1 > > > > *** Please download, test and verify on this release. This release > > candidate verification will stay open until Feb 2 *** > > > > Note that we are verifying upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.0-candidate-5/ > > > > SHA-512 checksums: > > > > 194b3a4d51b972ec58c8f2ae4ccaadb3cac229984ea5e7e8a396a1210d4b3adde83ab30ef31c9aa384942f81959da91ab250f5689cd010b4ae71a2b10956af2c > > > > apache-pulsar-3.2.0-bin.tar.gz > > > > 7248f2566627d772093204a61ae2ea87b58dd18f374d2fd624827eb95f4102bfe07b2be87a4e7b6c7c296c5791af623f308885bfbe89069d10620f9da73ded93 > > > > apache-pulsar-3.2.0-src.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1265/ > > > > The tag to verify: > > v3.2.0-candidate-5 (802576372132617b5076a44004846f2dbabede08) > > https://github.com/apache/pulsar/commits/v3.2.0-candidate-5/ > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Docker images: > > > > pulsar images: > > https://hub.docker.com/layers/technoboy8/pulsar/3.2.0-8025763/images/sha256-4666cc754439a2e6844569bb500365ded382b81d8fc9d4552e3c435702b59d86?context=repo > > <https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore> > > pulsar-all images: > > https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.0-8025763/images/sha256-b308fd819298bb2badc20ecd86547c43a7c8652aebd716816c7f8f24dbb1b34e?context=repo > > > > Please download the source package, and follow the README to build > > and run the Pulsar standalone service. > > > > Note that this RC doesn't require a formal vote, but we would also > > appreciate your feedback with +1/-1. And please provide specific > > comments if your feedback is not +1. > > > > > > > > Regards > > Jiwei Guo (Tboy) > >
[ANNOUNCE] Apache Pulsar Go Client 0.12.0 released
The Apache Pulsar team is proud to announce Apache Pulsar Go Client version 0.12.0. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar release details and downloads, visit: https://github.com/apache/pulsar-client-go/releases/tag/v0.12.0 Release Notes are at: https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG.md We would like to thank the contributors who made the release possible. Regards, The Pulsar Team
Re: [VOTE] Pulsar Client Go Release 0.12.0 Candidate 3
Thanks to everyone! Close this vote with 3 bindings +1s: - Penghui - Yunze - Yubiao Feng Thanks, Zike Yang On Fri, Jan 26, 2024 at 2:26 PM Yubiao Feng wrote: > > +1 (binding) > > - Build from source > - Run Pub & Sub with the pulsar cluster(branch-master) > > Thanks > Yubiao Feng > > On Wed, Jan 24, 2024 at 2:58 PM Zike Yang wrote: > > > Hi everyone, > > Please review and vote on the release candidate #3 for the version > > 0.12.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This is the third release candidate for Apache Pulsar Go client, > > version 0.12.0. > > > > The release note/changelog for Go client 0.12.0: > > https://github.com/apache/pulsar-client-go/pull/1153/files > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this > > release: > > https://downloads.apache.org/pulsar/KEYS > > > > Please download these packages and review this release candidate: > > - Review release notes: > > https://github.com/apache/pulsar-client-go/pull/1153 > > - Download the source package (verify shasum, and asc) and follow the > > README.md to build and run the pulsar-client-go. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source file: > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-3/ > > > > The tag to be voted upon: > > v0.12.0-candidate-3 > > https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-3 > > > > SHA-512 checksums: > > > > 66c37c4439549a02d7e3d21ecc02c4a2f44bad2232fbf15fb04c42f02fc4caad758b7a1b1663d0df47414af99daa4e3a19d33c29c0decbc43ac4d17e46a126bc > > apache-pulsar-client-go-0.12.0-src.tar.gz > >
Re: [DISCUSS] Release Pulsar Node.js client v1.10.0
+1 Thanks, Zike Yang On Mon, Jan 29, 2024 at 9:30 AM PengHui Li wrote: > > +1 > > Thanks for driving the release. > > Penghui > > On Mon, Jan 29, 2024 at 9:08 AM Baodi Shi wrote: > > > Hi all, > > > > I would like to propose releasing the Pulsar Node.js client v1.10.0 > > > > It has been over 5 months since the last release (1.9.0). There have > > been many new features and bug fixes since then. It's time to release > > a new version. > > > > > >- https://github.com/apache/pulsar-client-node/milestone/15?closed=1 > > > > > > Please let me know if you have any PRs that need to be included in 1.10.0. > > > > > > Thanks, > > Baodi Shi > >
Re: [VOTE] PIP-329: Strategy for maintaining the latest tag to Pulsar docker images
Thanks to everyone! Close this vote with 3 bindings +1s: - Jiwei Guo (Tboy) - Penghui - Yunze And 1 non-binding +1: - Zixuan Thanks, Zike Yang On Fri, Jan 19, 2024 at 11:05 AM Yunze Xu wrote: > > +1 (binding) > > Thanks, > Yunze > > On Wed, Jan 17, 2024 at 12:37 PM PengHui Li wrote: > > > > +1 (binding) > > > > Regards, > > Penghui > > > > On Wed, Jan 17, 2024 at 11:05 AM guo jiwei wrote: > > > > > +1 (binding) > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Mon, Jan 15, 2024 at 5:50 PM Zixuan Liu wrote: > > > > > > > +1(non-binding) > > > > > > > > Thanks, > > > > Zixuan > > > > > > > > Zike Yang 于2024年1月15日周一 16:14写道： > > > > > > > > > Hi, all > > > > > > > > > > I'm starting the vote for PIP-329: Strategy for maintaining the latest > > > > > tag to Pulsar docker images > > > > > > > > > > PIP link: https://github.com/apache/pulsar/pull/21872 > > > > > > > > > > Thanks, > > > > > Zike Yang > > > > > > > > > > > >
[VOTE] Pulsar Client Go Release 0.12.0 Candidate 3
Hi everyone, Please review and vote on the release candidate #3 for the version 0.12.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the third release candidate for Apache Pulsar Go client, version 0.12.0. The release note/changelog for Go client 0.12.0: https://github.com/apache/pulsar-client-go/pull/1153/files Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://downloads.apache.org/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1153 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-3/ The tag to be voted upon: v0.12.0-candidate-3 https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-3 SHA-512 checksums: 66c37c4439549a02d7e3d21ecc02c4a2f44bad2232fbf15fb04c42f02fc4caad758b7a1b1663d0df47414af99daa4e3a19d33c29c0decbc43ac4d17e46a126bc apache-pulsar-client-go-0.12.0-src.tar.gz
Re: [VOTE] Pulsar Client Go Release 0.12.0 Candidate 2
Hi yunze, Thanks for reporting this. Actually, the `VERSION` file is not used in the code even before that PR. We currently have two hardcoded version files in place: `VERSION` which represents the current version and `stable.txt` which represents the current stable version. However, I'm curious about the intended function of these files. Are they merely for developers to conveniently reference the current and stable versions? And what is our approach to managing these files? For instance, how do we determine what constitutes a stable version? If there's no compelling reason to keep these files, I'd support their removal. Regarding the current 0.12.0 release, before we make a consensus on whether we should remove these files. I can update the current version in those files in the next release candidate. Thanks, Zike Yang On Fri, Jan 19, 2024 at 4:27 PM Yunze Xu wrote: > > The VERSION file is still 0.11.1, which is not updated. But it seems > that this VERSION file is not used anywhere. I checked the stats and > found the "clientVersion" is "Pulsar Go v0.12.0-candidate-2", which is > expected. > > I see https://github.com/apache/pulsar-client-go/pull/856 supports > getting the version from the dependency, then the hard-coded VERSION > file is no longer needed. I think we can remove that file in the > 0.12.0 or next release. But we should not maintain a file that could > make users confused (i.e. the source code of 0.12.0 contains a version > file of 0.11.1) > > Thanks, > Yunze > > On Mon, Jan 15, 2024 at 2:41 PM Zike Yang wrote: > > > > Hi everyone, > > Please review and vote on the release candidate #2 for the version > > 0.12.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This is the second release candidate for Apache Pulsar Go client, > > version 0.12.0. > > > > The release note/changelog for Go client 0.12.0: > > https://github.com/apache/pulsar-client-go/pull/1153/files > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: > > https://downloads.apache.org/pulsar/KEYS > > > > Please download these packages and review this release candidate: > > - Review release notes: https://github.com/apache/pulsar-client-go/pull/1153 > > - Download the source package (verify shasum, and asc) and follow the > > README.md to build and run the pulsar-client-go. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source file: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-2/ > > > > The tag to be voted upon: > > v0.12.0-candidate-2 > > https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-2 > > > > SHA-512 checksums: > > e3c0845f60989b9ca4f6cf683d750c9361ca0e15ae7f8d990cfe14d7af86bd80475aeeda95870f2b8ef6fe8ec5fdbd59c67876546e282d76baae9a1737c6a7aa > > apache-pulsar-client-go-0.12.0-src.tar.gz
Re: [VOTE] PIP-330: getMessagesById gets all messages
+1 (non-binding) Thanks, Zike Yang On Mon, Jan 15, 2024 at 5:34 PM Zixuan Liu wrote: > > Hi Pulsar Community, > > Voting for PIP-330: getMessagesById gets all messages > > PIP: https://github.com/apache/pulsar/pull/21873 > Discussion thread: > https://lists.apache.org/thread/vqyh3mvtvovd383sd8zxnlzsspdr863z > > Thanks, > Zixuan
[VOTE] PIP-329: Strategy for maintaining the latest tag to Pulsar docker images
Hi, all I'm starting the vote for PIP-329: Strategy for maintaining the latest tag to Pulsar docker images PIP link: https://github.com/apache/pulsar/pull/21872 Thanks, Zike Yang
[VOTE] Pulsar Client Go Release 0.12.0 Candidate 2
Hi everyone, Please review and vote on the release candidate #2 for the version 0.12.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the second release candidate for Apache Pulsar Go client, version 0.12.0. The release note/changelog for Go client 0.12.0: https://github.com/apache/pulsar-client-go/pull/1153/files Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://downloads.apache.org/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1153 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-2/ The tag to be voted upon: v0.12.0-candidate-2 https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-2 SHA-512 checksums: e3c0845f60989b9ca4f6cf683d750c9361ca0e15ae7f8d990cfe14d7af86bd80475aeeda95870f2b8ef6fe8ec5fdbd59c67876546e282d76baae9a1737c6a7aa apache-pulsar-client-go-0.12.0-src.tar.gz
Re: [VOTE] Pulsar Client Go Release 0.12.0 Candidate 1
I will also include this PR to 0.12.0: https://github.com/apache/pulsar-client-go/pull/1156 It fixes a regression bug related to https://github.com/apache/pulsar/issues/21888. The bug is introduced from https://github.com/apache/pulsar-client-go/pull/1137 Thanks crossoverJie for the quick fix. BR, Zike Yang On Fri, Jan 12, 2024 at 4:04 PM Zike Yang wrote: > > > Could you include this fix > (https://github.com/apache/pulsar-client-go/pull/1155), which fixes a > CVE? > > Sure. I am closing this vote. And will start releasing candidate 2 later. > > Thanks, > Zike Yang > > On Fri, Jan 12, 2024 at 3:50 PM Yunze Xu wrote: > > > > Could you include this fix > > (https://github.com/apache/pulsar-client-go/pull/1155), which fixes a > > CVE? > > > > Thanks, > > Yunze > > > > On Wed, Jan 10, 2024 at 3:21 PM Zike Yang wrote: > > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the version > > > 0.12.0, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > This is the first release candidate for Apache Pulsar Go client, version > > > 0.12.0. > > > > > > The release note/changelog for Go client 0.12.0: > > > https://github.com/apache/pulsar-client-go/pull/1153/files > > > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this > > > release: > > > https://downloads.apache.org/pulsar/KEYS > > > > > > Please download these packages and review this release candidate: > > > - Review release notes: > > > https://github.com/apache/pulsar-client-go/pull/1153 > > > - Download the source package (verify shasum, and asc) and follow the > > > README.md to build and run the pulsar-client-go. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Source file: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-1/ > > > > > > The tag to be voted upon: > > > v0.12.0-candidate-1 > > > https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-1 > > > > > > SHA-512 checksums: > > > a32e1072646a13aa54f983f0d3fce9c1917773fcb261d3431e2ecf1bac519cf50d3e8c3479f527e3ec8fb69518e987f25f1e3b9eae6736739aaea77941766304 > > > apache-pulsar-client-go-0.12.0-src.tar.gz
Re: [VOTE] Pulsar Client Go Release 0.12.0 Candidate 1
> Could you include this fix (https://github.com/apache/pulsar-client-go/pull/1155), which fixes a CVE? Sure. I am closing this vote. And will start releasing candidate 2 later. Thanks, Zike Yang On Fri, Jan 12, 2024 at 3:50 PM Yunze Xu wrote: > > Could you include this fix > (https://github.com/apache/pulsar-client-go/pull/1155), which fixes a > CVE? > > Thanks, > Yunze > > On Wed, Jan 10, 2024 at 3:21 PM Zike Yang wrote: > > > > Hi everyone, > > Please review and vote on the release candidate #1 for the version > > 0.12.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This is the first release candidate for Apache Pulsar Go client, version > > 0.12.0. > > > > The release note/changelog for Go client 0.12.0: > > https://github.com/apache/pulsar-client-go/pull/1153/files > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: > > https://downloads.apache.org/pulsar/KEYS > > > > Please download these packages and review this release candidate: > > - Review release notes: https://github.com/apache/pulsar-client-go/pull/1153 > > - Download the source package (verify shasum, and asc) and follow the > > README.md to build and run the pulsar-client-go. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source file: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-1/ > > > > The tag to be voted upon: > > v0.12.0-candidate-1 > > https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-1 > > > > SHA-512 checksums: > > a32e1072646a13aa54f983f0d3fce9c1917773fcb261d3431e2ecf1bac519cf50d3e8c3479f527e3ec8fb69518e987f25f1e3b9eae6736739aaea77941766304 > > apache-pulsar-client-go-0.12.0-src.tar.gz
Re: [VERIFY] Pulsar Release 3.2.0 Candidate 1
There is a regression bug introduced in 3.2.0: https://github.com/apache/pulsar/issues/21888 The producer name will be conflicted when multiple consumers in the same topic and subscription send messages to the same DLQ concurrently. I will provide a fix later. Best, Zike Yang On Tue, Jan 9, 2024 at 8:34 PM houxiaoyu wrote: > > +1 (non-binding) > > > - Checksum and signatures > - Build on Mac, using JDK17.0.8 > - Run Pulsar standalone and produce/consume case > > Thanks, > Xiaoyu Hou > > guo jiwei 于2024年1月9日周二 16:15写道： > > > > This is the first release candidate for Apache Pulsar version 3.2.0. > > > > It fixes the following issues: > > https://github.com/apache/pulsar/milestone/36?closed=1 > > > > *** Please download, test and verify on this release. This release > > candidate verification will stay open until Jan 15 *** > > > > Note that we are verifying upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.2.0-candidate-1/ > > > > SHA-512 checksums: > > > > cf4e661d78f4194f4cde88e102bc9a734e2777613c5315f09115d64236759fbc5ded868c7e0cc3d07813c3fd34b1b123ca6f06f4335d6c9c96393789c6eda5bc > > > > apache-pulsar-3.2.0-bin.tar.gz > > > > 6ebe72de801db9a3f51dde39587b93d947904856385a21020319edb6988776fd68ec079471bcce5db17ea1b79c58e22ebf89a854debb40437ad9dc2bc2350357 > > > > apache-pulsar-3.2.0-src.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1259/ > > > > The tag to verify: > > v3.2.0-candidate-1 (4ab09374ade3c1402812c3c6be0d07984ab373a4) > > https://github.com/apache/pulsar/commits/v3.2.0-candidate-1/ > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Docker images: > > > > pulsar images: > > https://hub.docker.com/layers/technoboy8/pulsar/3.2.0-4ab0937/images/sha256-06d35d60f3bd4f954b8b1b6f4cc4f663556ace5da097173f41a61622e81a76b9?context=repo > > <https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore> > > pulsar-all images: > > https://hub.docker.com/layers/technoboy8/pulsar-all/3.2.0-4ab0937/images/sha256-a6f87bc5fd8025f096f35f26d47303267452e35d92f9f974d8536b61543ddbeb?context=repo > > > > Please download the source package, and follow the README to build > > and run the Pulsar standalone service. > > > > Note that this RC doesn't require a formal vote, but we would also > > appreciate your feedback with +1/-1. And please provide specific > > comments if your feedback is not +1. > > > > > > Regards > > Jiwei Guo (Tboy)
[DISCUSS] PIP-329: Strategy for maintaining the latest tag to Pulsar docker images
Hi, all I've created a new proposal to define the strategy for maintaining the latest tag to Pulsar docker images in the release process. Details and motivation in the PIP: https://github.com/apache/pulsar/pull/21872 Please help review it. Thanks, Zike Yang
[VOTE] Pulsar Client Go Release 0.12.0 Candidate 1
Hi everyone, Please review and vote on the release candidate #1 for the version 0.12.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the first release candidate for Apache Pulsar Go client, version 0.12.0. The release note/changelog for Go client 0.12.0: https://github.com/apache/pulsar-client-go/pull/1153/files Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://downloads.apache.org/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1153 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.12.0-candidate-1/ The tag to be voted upon: v0.12.0-candidate-1 https://github.com/apache/pulsar-client-go/tree/v0.12.0-candidate-1 SHA-512 checksums: a32e1072646a13aa54f983f0d3fce9c1917773fcb261d3431e2ecf1bac519cf50d3e8c3479f527e3ec8fb69518e987f25f1e3b9eae6736739aaea77941766304 apache-pulsar-client-go-0.12.0-src.tar.gz
[DISCUSS] Release Pulsar Go Client 0.12.0
Hi everyone, I would like to propose releasing the Pulsar Go Client 0.12.0. It has been several months since the last release. And there are several new features and bug fixes in the master branch[0]. We have also merged the pulsaradmin into the pulsar-client-go repo. It’s time to release a new version. Please let me know if you have any PRs that need to be included in 0.12.0. [0] https://github.com/apache/pulsar-client-go/compare/v0.11.0...master Thanks, Zike Yang
Re: [DISCUSS] Add a new set of Python Client APIs for asyncio and coroutines
LGTM. I'm +1 for making it as a submodule. I have approved the PR. Thanks, Zike Yang On Fri, Dec 29, 2023 at 6:33 PM Yunze Xu wrote: > > Hi all, > > Recently I noticed the Pulsar Python client APIs are not friendly to > Python users [1]. The existing asynchronous APIs are based on > callbacks and they are hard to use. Since the Pulsar Python client is > a wrapper on the Pulsar C++ client, wrapping the existing > callback-based APIs for asyncio requires much knowledge on the > underlying C++ implementation details [2]. > > For the reasons above, I think it will be helpful to provide another > set of APIs based on asyncio and coroutines. I opened my first PR [3] > as the beginning. Instead of rewriting a new project, I'd like to add > a submodule `pulsar.asyncio` for these APIs that share the same C > extension with APIs in the `pulsar` module. > > The new APIs are in development and do not affect the original APIs, > which are actually not tested well, see the regression [4]. Instead of > adding more tests to existing APIs, I'd like to pay more attention > when adding new APIs. Tests are required for each new configuration. > > Feel free to share your thoughts on this! > > [1] https://github.com/apache/pulsar-client-python/issues/184 > [2] > https://github.com/apache/pulsar-client-python/issues/55#issuecomment-1759925342 > [3] https://github.com/apache/pulsar-client-python/pull/189 > [4] https://github.com/apache/pulsar-client-python/issues/178 > > Thanks, > Yunze
[ANNOUNCE] Apache Pulsar Client Python 3.4.0 released
The Apache Pulsar team is proud to announce Apache Pulsar Client Python version 3.4.0. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. You can download the source code and the Python wheels in: https://archive.apache.org/dist/pulsar/pulsar-client-python-3.4.0/ The Python wheels were uploaded to PyPI as well so that they can be installed by `pip install pulsar-client==3.4.0`. Release Notes are at: https://pulsar.apache.org/release-notes/versioned/client-python-3.4.0/ We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
[ANNOUNCE] Apache Pulsar Client Python 3.4.0 released
The Apache Pulsar team is proud to announce Apache Pulsar Client Python version 3.4.0. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. You can download the source code and the Python wheels in: https://archive.apache.org/dist/pulsar/pulsar-client-python-3.4.0/ The Python wheels were uploaded to PyPI as well so that they can be installed by `pip install pulsar-client==3.4.0`. Release Notes are at: https://pulsar.apache.org/release-notes/versioned/client-python-3.4.0/ We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
Re: [VOTE] Pulsar Client Python Release 3.4.0 Candidate 1
Thanks, everyone! Close this vote with 3 binding votes: - Penghui - Yunze - Jiwei Guo (Tboy) Thanks, Zike Yang On Wed, Jan 3, 2024 at 2:30 PM guo jiwei wrote: > > +1 (binding) > > - Checked the signature and checksums > - Install the pulsar_client-3.4.0-cp39-cp39-macosx_10_15_universal2.whl > - Run Pulsar standalone > - Run the Python examples producer and consumer > > > Regards > Jiwei Guo (Tboy) > > > On Fri, Dec 29, 2023 at 12:34 PM Yunze Xu wrote: > > > +1 (binding) > > > > For source code, > > * verified signatures and checksums > > * built from source on macOS (with C++ client 3.4.2) > > > > For the following wheels, verified end-to-end with OAuth2 > > authentication on specific platforms. > > * the wheel built from source (macOS Ventura 13.6.3) > > * pulsar_client-3.4.0-cp312-cp312-macosx_10_15_universal2.whl (macOS > > Ventura 13.6.3) > > * > > pulsar_client-3.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > > (Rocky Linux 9) > > * > > pulsar_client-3.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl > > (Ubuntu 22.04) > > > > Thanks, > > Yunze > > > > On Thu, Dec 28, 2023 at 5:13 PM PengHui Li wrote: > > > > > > +1 (binding) > > > > > > - Checked the signature > > > - Install the pulsar_client-3.4.0-cp39-cp39-macosx_10_15_universal2.whl > > > - Run Pulsar standalone > > > - Run the Python examples `python3 ./examples/consumer.py` and `python3 > > > ./examples/producer.py` > > > > > > Regards, > > > Penghui > > > > > > On Tue, Dec 26, 2023 at 4:54 PM Zike Yang wrote: > > > > > > > This is the first release candidate for Apache Pulsar Client Python, > > > > version 3.4.0. > > > > > > > > It fixes the following issues: > > > > https://github.com/apache/pulsar-client-python/milestone/5?closed=1 > > > > > > > > *** Please download, test and vote on this release. This vote will > > > > stay open for at least 72 hours *** > > > > > > > > Python wheels: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.4.0-candidate-1/ > > > > > > > > The supported python versions are 3.8, 3.9, 3.10, 3.11 and 3.12. The > > > > supported platforms and architectures are: > > > > - Windows x86_64 (windows/) > > > > - glibc-based Linux x86_64 (linux-glibc-x86_64/) > > > > - glibc-based Linux arm64 (linux-glibc-arm64/) > > > > - musl-based Linux x86_64 (linux-musl-x86_64/) > > > > - musl-based Linux arm64 (linux-musl-arm64/) > > > > - macOS universal 2 (macos/) > > > > > > > > You can download the wheel (the `.whl` file) according to your own OS > > > > and Python version > > > > and install the wheel: > > > > - Windows: `py -m pip install *.whl --force-reinstall` > > > > - Linux or macOS: `python3 -m pip install *.whl --force-reinstall` > > > > > > > > The tag to be voted upon: v3.4.0-candidate-1 > > > > (v3.4.0-candidate-1) > > > > https://github.com/apache/pulsar-client-python/tree/v3.4.0-candidate-1 > > > > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > > > https://downloads.apache.org/pulsar/KEYS > > > > > > > > Please download the Python wheels and follow the README to test. > > > > > >
[VOTE] Pulsar Client Python Release 3.4.0 Candidate 1
This is the first release candidate for Apache Pulsar Client Python, version 3.4.0. It fixes the following issues: https://github.com/apache/pulsar-client-python/milestone/5?closed=1 *** Please download, test and vote on this release. This vote will stay open for at least 72 hours *** Python wheels: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.4.0-candidate-1/ The supported python versions are 3.8, 3.9, 3.10, 3.11 and 3.12. The supported platforms and architectures are: - Windows x86_64 (windows/) - glibc-based Linux x86_64 (linux-glibc-x86_64/) - glibc-based Linux arm64 (linux-glibc-arm64/) - musl-based Linux x86_64 (linux-musl-x86_64/) - musl-based Linux arm64 (linux-musl-arm64/) - macOS universal 2 (macos/) You can download the wheel (the `.whl` file) according to your own OS and Python version and install the wheel: - Windows: `py -m pip install *.whl --force-reinstall` - Linux or macOS: `python3 -m pip install *.whl --force-reinstall` The tag to be voted upon: v3.4.0-candidate-1 (v3.4.0-candidate-1) https://github.com/apache/pulsar-client-python/tree/v3.4.0-candidate-1 Pulsar's KEYS file containing PGP keys you use to sign the release: https://downloads.apache.org/pulsar/KEYS Please download the Python wheels and follow the README to test.
Re: [DISCUSS] Strategy for pushing the latest tag to Pulsar docker images
Hi, Enrico, > On the other side we should not encourage people to use 'latest' in production as because it is by design unpredictable. Agree. Fortunately, for the current document, we don't encourage people to use 'latest', We specify the specific version when starting the Pulsar docker in the doc. https://pulsar.apache.org/docs/next/getting-started-docker/ > Honestly it would be better to not have a 'latest' tag. Lack of the latest tag makes it less user-friendly and less intuitive for the user. For most users who are new to Pulsar, they can just start the Pulsar container without having to know beforehand what version of Pulsar is out there. I think we still need the latest tag. Just want to make sure do you +1 for this PR? https://github.com/apache/pulsar-site/pull/745 Thanks, Zike Yang On Tue, Dec 12, 2023 at 7:50 PM Enrico Olivelli wrote: > > The first option is better, as it allows people to try out the latest > features and hopefully it is more stable. > > On the other side we should not encourage people to use 'latest' in > production as because it is by design unpredictable. Let's day that you > have a system that had been running for a year, then you restart it and the > system picks a new major release (say from 3.0 to 4 > 0), then you have no control over the new defaults or about dropped > features. > > Honestly it would be better to not have a 'latest' tag. > > Enrico > > Il Mar 12 Dic 2023, 08:41 Zike Yang ha scritto: > > > Hi, penghui > > > > > Is it better to introduce a `lts` or `lts-latest` tag to the images? > > > > Good idea. I would prefer the `lts` tag. I have updated the PR for this. > > > > Thanks, > > Zike Yang > > > > On Tue, Dec 12, 2023 at 11:43 AM PengHui Li wrote: > > > > > > +1 for the first way. > > > > > > Is it better to introduce a `lts` or `lts-latest` tag to the images? > > > Users can use the new tag if they want to get the latest LTS version. > > > > > > Thanks, > > > Penghui > > > > > > > > > On Tue, Dec 12, 2023 at 10:09 AM Zike Yang wrote: > > > > > > > Thanks to all of you. > > > > > > > > I believe we have reached an agreement on this. Could you help review > > > > the PR? https://github.com/apache/pulsar-site/pull/745 > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > On Thu, Dec 7, 2023 at 12:11 PM Zixuan Liu wrote: > > > > > > > > > > Hi Zike, > > > > > > > > > > > I prefer the 1st way as well. > > > > > +1 > > > > > > > > > > Thanks, > > > > > Zixuan > > > > > > > > > > Yunze Xu 于2023年12月5日周二 20:26写道： > > > > > > > > > > > I prefer the 1st way as well. > > > > > > > > > > > > Thanks, > > > > > > Yunze > > > > > > > > > > > > On Tue, Dec 5, 2023 at 5:11 PM Zike Yang wrote: > > > > > > > > > > > > > > Hi, everyone, > > > > > > > > > > > > > > There seems to be a gap in our current release process > > concerning the > > > > > > > pushing of the latest tag to our Docker images. Specifically, we > > need > > > > > > > to decide which version of Pulsar the latest tag should point to. > > > > > > > > > > > > > > I have come up with two options: > > > > > > > 1. The latest tag could point to the most recent feature release > > or > > > > > > > the subsequent patch of that feature release. For instance, it > > could > > > > > > > currently point to 3.1.1, and in the future, it could point to > > 3.1.2 > > > > > > > or 3.2.0. > > > > > > > 2. The latest tag could point to the most recent LTS release or > > the > > > > > > > subsequent patch of that LTS release. As an example, it could > > > > > > > currently point to 3.0.2, and in the future, it could point to > > 3.0.3 > > > > > > > or 4.0.0. > > > > > > > > > > > > > > What do you think? I would prefer the option 1. I have pushed a > > PR to > > > > > > > update the release docker image process: > > > > > > > https://github.com/apache/pulsar-site/pull/745. > > > > > > > PTAL and leave your comments and feedback here. > > > > > > > > > > > > > > Thanks, > > > > > > > Zike Yang > > > > > > > > > > > >
Re: [DISCUSS] Strategy for pushing the latest tag to Pulsar docker images
Hi, penghui > Is it better to introduce a `lts` or `lts-latest` tag to the images? Good idea. I would prefer the `lts` tag. I have updated the PR for this. Thanks, Zike Yang On Tue, Dec 12, 2023 at 11:43 AM PengHui Li wrote: > > +1 for the first way. > > Is it better to introduce a `lts` or `lts-latest` tag to the images? > Users can use the new tag if they want to get the latest LTS version. > > Thanks, > Penghui > > > On Tue, Dec 12, 2023 at 10:09 AM Zike Yang wrote: > > > Thanks to all of you. > > > > I believe we have reached an agreement on this. Could you help review > > the PR? https://github.com/apache/pulsar-site/pull/745 > > > > Thanks, > > Zike Yang > > > > On Thu, Dec 7, 2023 at 12:11 PM Zixuan Liu wrote: > > > > > > Hi Zike, > > > > > > > I prefer the 1st way as well. > > > +1 > > > > > > Thanks, > > > Zixuan > > > > > > Yunze Xu 于2023年12月5日周二 20:26写道： > > > > > > > I prefer the 1st way as well. > > > > > > > > Thanks, > > > > Yunze > > > > > > > > On Tue, Dec 5, 2023 at 5:11 PM Zike Yang wrote: > > > > > > > > > > Hi, everyone, > > > > > > > > > > There seems to be a gap in our current release process concerning the > > > > > pushing of the latest tag to our Docker images. Specifically, we need > > > > > to decide which version of Pulsar the latest tag should point to. > > > > > > > > > > I have come up with two options: > > > > > 1. The latest tag could point to the most recent feature release or > > > > > the subsequent patch of that feature release. For instance, it could > > > > > currently point to 3.1.1, and in the future, it could point to 3.1.2 > > > > > or 3.2.0. > > > > > 2. The latest tag could point to the most recent LTS release or the > > > > > subsequent patch of that LTS release. As an example, it could > > > > > currently point to 3.0.2, and in the future, it could point to 3.0.3 > > > > > or 4.0.0. > > > > > > > > > > What do you think? I would prefer the option 1. I have pushed a PR to > > > > > update the release docker image process: > > > > > https://github.com/apache/pulsar-site/pull/745. > > > > > PTAL and leave your comments and feedback here. > > > > > > > > > > Thanks, > > > > > Zike Yang > > > > > >
Re: [DISCUSS] Strategy for pushing the latest tag to Pulsar docker images
Thanks to all of you. I believe we have reached an agreement on this. Could you help review the PR? https://github.com/apache/pulsar-site/pull/745 Thanks, Zike Yang On Thu, Dec 7, 2023 at 12:11 PM Zixuan Liu wrote: > > Hi Zike, > > > I prefer the 1st way as well. > +1 > > Thanks, > Zixuan > > Yunze Xu 于2023年12月5日周二 20:26写道： > > > I prefer the 1st way as well. > > > > Thanks, > > Yunze > > > > On Tue, Dec 5, 2023 at 5:11 PM Zike Yang wrote: > > > > > > Hi, everyone, > > > > > > There seems to be a gap in our current release process concerning the > > > pushing of the latest tag to our Docker images. Specifically, we need > > > to decide which version of Pulsar the latest tag should point to. > > > > > > I have come up with two options: > > > 1. The latest tag could point to the most recent feature release or > > > the subsequent patch of that feature release. For instance, it could > > > currently point to 3.1.1, and in the future, it could point to 3.1.2 > > > or 3.2.0. > > > 2. The latest tag could point to the most recent LTS release or the > > > subsequent patch of that LTS release. As an example, it could > > > currently point to 3.0.2, and in the future, it could point to 3.0.3 > > > or 4.0.0. > > > > > > What do you think? I would prefer the option 1. I have pushed a PR to > > > update the release docker image process: > > > https://github.com/apache/pulsar-site/pull/745. > > > PTAL and leave your comments and feedback here. > > > > > > Thanks, > > > Zike Yang > >
Re: [ANNOUNCE] Apache Pulsar 3.0.2 released
Hi, Kiryl Thanks for reporting. I have submitted a PR to fix it: https://github.com/apache/pulsar-site/pull/748 Thanks, Zike Yang On Fri, Dec 8, 2023 at 2:36 PM Kiryl Valkovich wrote: > > Hi, Zike > > Thanks for fixing it. > > One more thing to fix. v3.0.2 release is missing on the Downloads page. I > raised one more issue here: > https://github.com/apache/pulsar/issues/21694 > > > From: Zike Yang > Date: Wednesday, December 6, 2023 at 1:08 PM > To: dev@pulsar.apache.org > Subject: Re: [ANNOUNCE] Apache Pulsar 3.0.2 released > Hi, Kiryl > > Thanks for your catch. I have submitted a PR to fix it: > https://github.com/apache/pulsar-site/pull/746 PTAL. > > Thanks, > Zike Yang > > On Wed, Dec 6, 2023 at 4:41 PM Kiryl Valkovich > wrote: > > > > Congrats on the release! > > > > Small fixes in the Release table are needed. I raised the issue here: > > https://github.com/apache/pulsar/issues/21677 > > > > > > Best, > > Kiryl > > > > From: Yubiao Feng > > Date: Wednesday, December 6, 2023 at 8:17 AM > > To: dev@pulsar.apache.org , us...@pulsar.apache.org > > , annou...@apache.org > > Subject: [ANNOUNCE] Apache Pulsar 3.0.2 released > > The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.2. > > > > Pulsar is a highly scalable, low-latency messaging platform running on > > commodity hardware. It provides simple pub-sub semantics over topics, > > guaranteed at-least-once delivery of messages, automatic cursor management > > for > > subscribers, and cross-datacenter replication. > > > > For Pulsar release details and downloads, visit: > > > > https://pulsar.apache.org/download > > > > Release Notes are at: > > https://pulsar.apache.org/release-notes > > > > We would like to thank the contributors that made the release possible. > > > > Regards, > > > > The Pulsar Team
Re: [ANNOUNCE] Apache Pulsar 3.0.2 released
Hi, Kiryl Thanks for your catch. I have submitted a PR to fix it: https://github.com/apache/pulsar-site/pull/746 PTAL. Thanks, Zike Yang On Wed, Dec 6, 2023 at 4:41 PM Kiryl Valkovich wrote: > > Congrats on the release! > > Small fixes in the Release table are needed. I raised the issue here: > https://github.com/apache/pulsar/issues/21677 > > > Best, > Kiryl > > From: Yubiao Feng > Date: Wednesday, December 6, 2023 at 8:17 AM > To: dev@pulsar.apache.org , us...@pulsar.apache.org > , annou...@apache.org > Subject: [ANNOUNCE] Apache Pulsar 3.0.2 released > The Apache Pulsar team is proud to announce Apache Pulsar version 3.0.2. > > Pulsar is a highly scalable, low-latency messaging platform running on > commodity hardware. It provides simple pub-sub semantics over topics, > guaranteed at-least-once delivery of messages, automatic cursor management > for > subscribers, and cross-datacenter replication. > > For Pulsar release details and downloads, visit: > > https://pulsar.apache.org/download > > Release Notes are at: > https://pulsar.apache.org/release-notes > > We would like to thank the contributors that made the release possible. > > Regards, > > The Pulsar Team
[DISCUSS] Strategy for pushing the latest tag to Pulsar docker images
Hi, everyone, There seems to be a gap in our current release process concerning the pushing of the latest tag to our Docker images. Specifically, we need to decide which version of Pulsar the latest tag should point to. I have come up with two options: 1. The latest tag could point to the most recent feature release or the subsequent patch of that feature release. For instance, it could currently point to 3.1.1, and in the future, it could point to 3.1.2 or 3.2.0. 2. The latest tag could point to the most recent LTS release or the subsequent patch of that LTS release. As an example, it could currently point to 3.0.2, and in the future, it could point to 3.0.3 or 4.0.0. What do you think? I would prefer the option 1. I have pushed a PR to update the release docker image process: https://github.com/apache/pulsar-site/pull/745. PTAL and leave your comments and feedback here. Thanks, Zike Yang
[DISCUSS] Release Pulsar Python clieng 3.4.0
Hi all, I would like to propose releasing the Pulsar Python client 3.4.0. It's over three months since the release of 3.3.0, and there are 46 new commits: https://github.com/apache/pulsar-client-python/compare/v3.0.0...main The Pulsar C++ client 3.4.1 has been released. I have upgraded the C++ client for the Python client to 3.4.1: https://github.com/apache/pulsar-client-python/pull/167 We need to cut a new release. Please let me know if you have any important fixes that need to be included in Pulsar Python client 3.4.0. Thanks, Zike Yang
Re: Pulsar 3.0.2 docker image only supports arm64 platform
We need to use the following commands to push the images: ```sh regctl image copy 9947090/pulsar:3.0.2-12c92fe apachepulsar/pulsar:3.0.2 regctl image copy 9947090/pulsar-all:3.0.2-12c92fe apachepulsar/pulsar-all:3.0.2 ``` It's missing in the release doc. I will update it later. Thanks, Zike Yang On Mon, Dec 4, 2023 at 10:31 AM Yunze Xu wrote: > > I found Matteo has updated 3.1.1 as the latest image: > https://hub.docker.com/layers/apachepulsar/pulsar/latest/images/sha256-21e8bf1571e4ab559a51b3f6e524d725cffabe3c6836101f9d7ea7eb1e2bf62c?context=explore > > But the 3.0.2 still lacks the image for the amd64 platform. > > Thanks, > Yunze > > On Mon, Dec 4, 2023 at 12:51 AM Yunze Xu wrote: > > > > Hi all, > > > > When I ran unit tests for the pulsar-client-cpp repo, which uses > > `apachepulsar/pulsar:latest` image to start a standalone, I found the > > tests failed and the following error [1]: > > > > ``` > > standalone The requested image's platform (linux/arm64) does not match > > the detected host platform (linux/amd64/v3) and no specific platform > > was requested > > ``` > > > > It seems the 3.0.2 image [2] was only built for the arm64 platform. I > > think there is something wrong with the steps to build the image. It > > breaks the CI for pulsar-client-cpp repo, as well as any other repo > > that depends on the latest image. We should fix the image and push it > > again. > > > > [1] > > https://github.com/apache/pulsar-client-cpp/actions/runs/7078043224/job/19263093582?pr=360 > > [2] > > https://hub.docker.com/layers/apachepulsar/pulsar/3.0.2/images/sha256-5af72da140f3901bccc0a5b56de8764cd60f0d351011fd1b90c484a30889fbf8?context=explore > > > > Thanks, > > Yunze
Re: [VOTE] Pulsar Release 3.0.2 Candidate 4
Hi Yubiao, I see that there is only arm64 arch for the docker image. We need to follow this guide to build the image: https://github.com/apache/pulsar-site/pull/595. The current release process is incorrect for building the docker image. This PR(https://github.com/apache/pulsar-site/pull/595) has been blocked for some time. We need to move this PR forward. BR, Zike Yang On Mon, Nov 20, 2023 at 5:07 PM Yubiao Feng wrote: > > Sorry, the SHA-512 checksums are wrong by mistake. Correct it below: > > apache-pulsar-3.0.2-bin.tar.gz > > b3421eb57233638e4d8305b040b1ffd1374b9ddfa55ea1a1043dc9be78ed89f72604cf249fcf5e663f1e3e1be86daf36991ee069ea1eb146f384c374484da0ff > > apache-pulsar-3.0.2-src.tar.gz > > 02154a2c190ac2ef7e1ff3fce5e90eab896f392317c28b28e9a6ab912a5ca3b3a38dc358b5925153ba1b0dde340cafd1e6aa72349dd9c9ae5dd99d0bdef587fb > > > Thanks > Yubiao Feng > > 233638e4d8305b040b1ffd1374b9ddfa55ea1a1043dc9be78ed89f72604cf249fcf5e663f1e3e1be86daf36991ee069ea1eb146f384c374484da0ff > ./apache-pulsar-3.0.2-bin.tar.gz > > On Mon, Nov 20, 2023 at 10:24 AM Yubiao Feng > wrote: > > > This is the first release candidate for Apache Pulsar version 3.0.4. > > > > It fixes the following issues: > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+is%3Amerged+label%3Arelease%2F3.0.2+label%3Acherry-picked%2Fbranch-3.0+ > > > > *** Please download, test and vote on this release. This vote will > > stay open for at least 72 hours *** > > > > Note that we are voting upon the source (tag), binaries are provided > > for convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.0.2-candidate-4/ > > > > SHA-512 checksums: > > > > > > 9bd5b4030de47c10a906d6271731ca7bcfb8801984cdd625789bc0c5047138f0f2eb1aaceaec1c8941e2ccfa639bbb2458496518e87fb29abb9ebc518e2da60d > > > > apache-pulsar-3.0.2-bin.tar.gz > > > > > > 6854d1776de08c9079228a5a5d2f670f92b1b905f0d92c50ee35c5bc9c53be5225626a6d84fc3bbb209b8b12ff3b142685b5c9ea33609e1bb934e9679402e0b8 > > > > apache-pulsar-3.0.2-src.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1249 > > > > The tag to verify: > > v3.0.2-candidate-4 (12c92fed7847965e3bc3769a91c866b2f0ec2e44) > > https://github.com/apache/pulsar/releases/tag/v3.0.2-candidate-4 > > > > Pulsar's KEYS file containing PGP keys you use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Docker images: > > > > pulsar images: > > https://hub.docker.com/repository/docker/9947090/pulsar-all > > > > pulsar-all images: > > https://hub.docker.com/repository/docker/9947090/pulsar > > > > Please download the source package, and follow the README to build > > and run the Pulsar standalone service. > > > > > > Regards > > Yubiao Feng(poorbarcode) > >
Re: [Discuss] Release Pulsar C++ Client 3.4.1
+1 Thanks for your fixes. BR, Zike Yang On Thu, Nov 16, 2023 at 5:48 PM Yunze Xu wrote: > > There is another critical issue introduced by 3.4.0: > https://github.com/apache/pulsar-client-python/issues/162 > > I will open another fix for it. > > Thanks, > Yunze > > On Thu, Nov 16, 2023 at 12:45 PM Yunze Xu wrote: > > > > In the latest C++ Client 3.4.0 release, we observed a regression > > https://github.com/apache/pulsar-client-cpp/issues/346. It's a > > critical bug that a crash could happen for each reconnection. > > > > I pushed a fix (https://github.com/apache/pulsar-client-cpp/pull/347). > > After it's merged, I will start the release process for 3.4.1. > > > > Please let me know if you have any questions. > > > > Thanks, > > Yunze
Re: [VOTE] PIP-312 Use StateStoreProvider to manage state in Pulsar Functions endpoints
+1(no-binding) Thanks, Zike Yang On Wed, Nov 15, 2023 at 4:01 PM Baodi Shi wrote: > > +1(non-binding) > > > Thanks, > Baodi Shi > > > On Nov 15, 2023 at 11:49:24, Neng Lu wrote: > > > +1 > > > > On 2023/11/15 03:39:42 Pengcheng Jiang wrote: > > > > Hi Pulsar Community, > > > > > > This thread is to start a vote for PIP-312: Use StateStoreProvider to > > > > manage state in Pulsar Functions endpoints. > > > > > > I start the voting process since there are some approves for the PIP PR. > > > > > > PR: https://github.com/apache/pulsar/pull/21438 > > > > Discussion thread: > > > > https://lists.apache.org/thread/0rz29wotonmdck76pdscwbqo19t3rbds > > > > > > Sincerely, > > > > Pengcheng Jiang > > > > > >
Re: [ANNOUNCE] Yubiao Feng as new PMC member in Apache Pulsar
Congratulations! BR, Zike Yang On Tue, Nov 14, 2023 at 3:34 AM Heesung Sohn wrote: > > Congrats! > Heesung > > On Mon, Nov 13, 2023 at 6:57 AM Yubiao Feng > wrote: > > > Thank you all > > > > Yubiao Feng > > > > > > > > On Mon, Nov 13, 2023 at 3:36 PM mattison chao > > wrote: > > > > > Dear Community, > > > > > > We are thrilled to announce that Yubiao Feng > > > https://github.com/poorbarcode has been invited and has accepted the > > role > > > of a member of the Apache Pulsar Project Management Committee (PMC). > > > > > > Yubiao Feng has proven to be an invaluable asset to our community, > > > consistently showcasing dedication and active engagement through > > > substantial contributions. Beyond his noteworthy technical input, Yubiao > > > plays a crucial role in meticulously reviewing pull requests, thereby > > > ensuring the overall excellence of our project. We eagerly anticipate and > > > appreciate his ongoing contributions. On behalf of the Pulsar PMC, we > > > extend a heartfelt welcome and congratulations to Yubiao Feng. > > > > > > Sincerely, > > > Mattison > >
[ANNOUNCE] Apache Pulsar Go Client 0.11.1 released
The Apache Pulsar team is proud to announce Apache Pulsar Go Client version 0.11.1. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar release details and downloads, visit: https://github.com/apache/pulsar-client-go/releases/tag/v0.11.1 Release Notes are at: https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG.md We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
Re: [VOTE] Pulsar Client Go Release 0.11.1 Candidate 1
Close this vote by 3 binding +1: - Yunze - Penghui - Mattison BR, Zike Yang On Tue, Nov 7, 2023 at 4:43 PM Zike Yang wrote: > > > The KEYS file link is dead, I think it should be > https://downloads.apache.org/pulsar/KEYS > > Thanks for your reminder. Yes, that should be the correct link. I have > also pushed a PR to fix the release process: > https://github.com/apache/pulsar-client-go/pull/1127 PTAL. Thanks! > > BR, > Zike Yang > > On Tue, Nov 7, 2023 at 4:00 PM mattison chao wrote: > > > > +1 (binding) > > > > - Built from the source > > - Ran the test on pulsar 3.0.1 > > > > Best, > > Mattison > > > > > On Sep 11, 2023, at 18:07, Zike Yang wrote: > > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the version > > > 0.11.1, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > This is the first release candidate for Apache Pulsar Go client, version > > > 0.11.1. > > > > > > It fixes the following issues: > > > https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1 > > > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this > > > release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > Please download these packages and review this release candidate: > > > - Review release notes: > > > https://github.com/apache/pulsar-client-go/pull/1092 > > > - Download the source package (verify shasum, and asc) and follow the > > > README.md to build and run the pulsar-client-go. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Source file: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/ > > > > > > The tag to be voted upon: > > > v0.11.1-candidate-1 > > > https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1 > > > > > > SHA-512 checksums: > > > d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182 > > > apache-pulsar-client-go-0.11.1-src.tar.gz > >
Re: [VOTE] Pulsar Client Go Release 0.11.1 Candidate 1
> The KEYS file link is dead, I think it should be https://downloads.apache.org/pulsar/KEYS Thanks for your reminder. Yes, that should be the correct link. I have also pushed a PR to fix the release process: https://github.com/apache/pulsar-client-go/pull/1127 PTAL. Thanks! BR, Zike Yang On Tue, Nov 7, 2023 at 4:00 PM mattison chao wrote: > > +1 (binding) > > - Built from the source > - Ran the test on pulsar 3.0.1 > > Best, > Mattison > > > On Sep 11, 2023, at 18:07, Zike Yang wrote: > > > > Hi everyone, > > Please review and vote on the release candidate #1 for the version > > 0.11.1, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This is the first release candidate for Apache Pulsar Go client, version > > 0.11.1. > > > > It fixes the following issues: > > https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1 > > > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Please download these packages and review this release candidate: > > - Review release notes: https://github.com/apache/pulsar-client-go/pull/1092 > > - Download the source package (verify shasum, and asc) and follow the > > README.md to build and run the pulsar-client-go. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source file: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/ > > > > The tag to be voted upon: > > v0.11.1-candidate-1 > > https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1 > > > > SHA-512 checksums: > > d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182 > > apache-pulsar-client-go-0.11.1-src.tar.gz >
Re: Question about Pulsar gRPC client(s)
Another point is that there are many features implemented on the client side, including batching, chunking, DLQ, etc. This makes it hard to replace the existing pulsar clients completely. Zike Yang On Wed, Nov 1, 2023 at 4:43 AM Christophe Bornet wrote: > > Hi Kiryl, > > Thanks for mentioning pulsar-grpc. > Indeed, using gRPC simplifies the implementation of the networking logic > (keep-alive, reconnection, flow control,…). On the other hand, the Java > gRPC implementation makes a lot of buffer copies to cleanly separate the > network and app layers but that takes a toll on performance. Compared to > that, the broker Pulsar protocol impl is optimized to not do copies between > the consumer/producer endpoints and the bookkeeper client. > So I think we could not replace completely the Pulsar protocol by gRPC. > We could maybe support both but it’s a lot of work to maintain both > protocols. (I kind of gave up maintaining pulsar-grpc because of the amount > of work compared to the number of users, but if there’s interest I can > reconsider). > Another possibility would be to do a proxy instead of a low-level protocol > handler. A bit like the WebSocket proxy. This would be far less work to > maintain as it would use the Pulsar protocol to communicate with the > brokers. It could be done as a Proxy extension. Compared to the WS proxy, > this would provide type safety, discovery, and so on… > As for the Admin, it’s a bit the same. It would be a bunch of work to > support both gRPC and REST. You have some kind of type hinting with the > OpenAPI spec that you can use to generate client SDKs (eg. with > openapi-generator. > I wonder what others have to say. > > Christophe > > > Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich > a écrit : > > > Hi! Am I understanding it right, that if this project > > https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar > > repo, we could easily cover non-mainstream platforms that are supported by > > gRPC, but don't have ready-to-use Pulsar clients? > > > > https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler > > > > Similar to already supported language-agnostic client interfaces - REST > > and WebSocket. > > > > Actively maintained gRPC libraries I found (19, or 15 if considering JVM > > languages and web as duplicates): > > - [C# / .NET](https://grpc.io/docs/languages/csharp/) > > - [C++](https://grpc.io/docs/languages/cpp/) > > - [Dart](https://grpc.io/docs/languages/dart/) > > - [Go](https://grpc.io/docs/languages/go/) > > - [Java](https://grpc.io/docs/languages/java/) > > - [Kotlin](https://grpc.io/docs/languages/kotlin/) > > - [Node](https://grpc.io/docs/languages/node/) > > - [Objective-C](https://grpc.io/docs/languages/objective-c/) > > - [PHP](https://grpc.io/docs/languages/php/) > > - [Python](https://grpc.io/docs/languages/python/) > > - [Ruby](https://grpc.io/docs/languages/ruby/) > > - [OCaml](https://github.com/dialohq/ocaml-grpc) > > - [Haskell](https://github.com/awakesecurity/gRPC-haskell) > > - [Elixir](https://github.com/elixir-grpc/grpc) > > - [Rust](https://github.com/hyperium/tonic) > > - [Scala](https://scalapb.github.io/) > > - [Swift](https://github.com/grpc/grpc-swift) > > - Web client: https://github.com/grpc/grpc-web > > - Web client 2: https://connectrpc.com/docs/web/getting-started > > > > Actively maintained Pulsar libraries (8): > > - Java > > - C++ > > - Python > > - Go > > - Node.js > > - C# > > - PHP > > - Rust > > > > Is there any reason for not merging it to the apache/pulsar? > > > > I would definitely prefer to work with a statically typed gRPC client > > instead of REST or WebSocket. > > > > By the way, the same can work for the Pulsar Admin API. Implement the gRPC > > server once in Java, and we have full-featured native statically typed > > (where applicable :)) Pulsar Admin clients for any platform. > > > > Best, > > Kiryl > >
Re: [DISCUSS] Auto release Nuget artifacts for DotPulsar
Hi, David Thanks for initializing this discussion. I'm +1 for making the auto-release process for the DotPulsar. I have left some comments in the PR. Thanks, Zike Yang On Thu, Oct 26, 2023 at 5:39 PM David Jensen wrote: > > Hi everyone, > I have worked with Tison to make the release process smoother for DotPulsar > <https://github.com/apache/pulsar-dotpulsar>. See GitHub Issue #184 Release > process and GitHub Action > <https://github.com/apache/pulsar-dotpulsar/pull/184> and the linked issues. > > In summary, the document presents a suggested approach for creating release > notes and optimizing the release workflow of the Pulsar C# Client. This > approach involves working closely with team members, adhering to ASF > guidelines, participating in Apache release voting, implementing automation > via GitHub Actions, and managing versioning through git tags and GitHub > releases. > > Sincerely > David Jensen
Re: [Discuss] Release Pulsar C++ Client 3.4.0
+1, BR, Zike Yang On Mon, Oct 23, 2023 at 1:10 PM PengHui Li wrote: > > Thanks for driving the release, > > +1 > > Penghui > > On Mon, Oct 23, 2023 at 11:00 AM Yunze Xu wrote: > > > I would like to propose releasing the Pulsar C++ Client 3.4.0. It has > > been about 3 months since the last release. There have been many new > > features and bug fixes since then. > > > > Besides, from my own perspective, it's better to let Python and > > Node.js clients depend on this new version of C++ client. Especially I > > observed that the topic name was not shown correctly many times > > recently, which are fixed by > > https://github.com/apache/pulsar-client-cpp/pull/331 and > > https://github.com/apache/pulsar-client-cpp/pull/329. > > > > So it's time to release a new version. Please let me know if you have > > any PRs that need to be included in 3.4.0 > > > > Thanks, > > Yunze > >
Re: [VOTE] PIP-302 Introduce refreshAsync API for TableView
+1 BR, Zike Yang On Wed, Sep 27, 2023 at 3:05 PM Xiangying Meng wrote: > > Hi dev, >This thread is to start a vote for PIP-302 Add new API > refreshAsync for TableView. > Discuss thread: > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q > PIP: https://github.com/apache/pulsar/pull/21166 > > BR, > Xiangying
Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins
Hi, Zixuan Thanks for your proposal. I'm +1 for it. > This is a feature I need. If cherry-pick is not allowed, then it will only be kept in 3.2+. This is a new feature, and I also think that we couldn't cherry-pick it. What about cherry-picking this change to your fork repo and building the pulsar for your own to meet this need? Does it make sense to you? BR, Zike Yang On Mon, Sep 25, 2023 at 12:23 AM mattison chao wrote: > > Hi, Zixuan > > I am afraid I can't support you in cherry-picking this feature for all of the > active branches by the current fact. Because this is a new feature and it's > not a bug fix or security issue. > > We can wait for other contributor's ideas. WDYT? > > Thanks, > Mattison > On 19 Sep 2023 at 10:42 +0800, Zixuan Liu , wrote: > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for all > > > branches after the target branch. Isn't it? > > > > I think we should cherry-pick this PR to Pulsar 2.10, 2.11, 3.0. > > > > > 2. I've got a slight concern about cherry-picking. Do you have any issue > > > or feature request in the community that can demonstrate this feature is > > > required to cherry-pick? > > > > This is a feature I need. If cherry-pick is not allowed, then it will > > only be kept in 3.2+. > > > > Thanks, > > Zixuan > > > > mattison chao 于2023年9月18日周一 09:42写道： > > > > > > > > HI, Zixuan > > > > > > Thanks for your discussion. It's a great feature. But I've got several > > > questions I want to discuss here. > > > > > > 1. When you want to cherry-pick a PR, it should be cherry-picked for all > > > branches after the target branch. Isn't it? > > > 2. I've got a slight concern about cherry-picking. Do you have any issue > > > or feature request in the community that can demonstrate this feature is > > > required to cherry-pick? > > > > > > > > > Best, > > > Mattison > > > On 12 Sep 2023 at 11:25 +0800, Zixuan Liu , wrote: > > > > > BTW, I think we can cherry-pick this feature to the Pulsar 2.10. As > > > > > far as I know, the Pulsar 2.10 is a stable/main version, because many > > > > > users are using it. > > > > > > > > > > Thanks, > > > > > Zixuan > > > > > > > > > > Zixuan Liu 于2023年9月5日周二 17:09写道： > > > > > > > > > > > > > > Hi Pulsar Community, > > > > > > > > > > > > > > Discuss for PIP-300: https://github.com/apache/pulsar/pull/21127 > > > > > > > > > > > > > > Thanks, > > > > > > > Zixuan
Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar
> If we want the maximum concurrency per second to be 1, and set the maximum concurrency per minute to 60, then the actual maximum concurrency per second could be up to 60, which is 60 times larger than our expected maximum concurrency. I think the RateLimiter can handle it: https://github.com/apache/pulsar/blob/a1405ea006f175b1bd0b9d28b9444d592fb4a010/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L965-L968 > Secondly, we already have the `maxConcurrentUnloadPerSec` configuration, which is provided to the user in the CLI. Adding a similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` configuration might confuse users. `maxConcurrentUnloadPerSec ` is for the admin and CLI usage. This proposal is to add `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute ` to the broker configuration. Overall, I'm +1 for this proposal. And I agree that we need a new PIP for this change. BR, Zike Yang On Mon, Sep 25, 2023 at 3:54 PM Xiangying Meng wrote: > > Hi Donglai, Heesung > > >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as > brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min" > config can be more granular. > > I have some doubts about introducing the > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > configuration. > > Firstly, I also think that a minute is too long. If we want the > maximum concurrency per second to be 1, and set the maximum > concurrency per minute to 60, then the actual maximum concurrency per > second could be up to 60, which is 60 times larger than our expected > maximum concurrency. Moreover, if the unload requests are concentrated > in the last 10 seconds of the previous minute and the first 10 seconds > of the next minute, then the concurrency during this period will > exceed our configuration. Such fluctuations are inevitable, but the > larger the time span we set, the greater the distortion of the > configuration. > > Secondly, we already have the `maxConcurrentUnloadPerSec` > configuration, which is provided to the user in the CLI. Adding a > similar `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > configuration might confuse users. When designing configuration > parameters, we should try to keep it simple and consistent, and avoid > introducing unnecessary complexity. > > Thanks, > Xiangying > > On Mon, Sep 25, 2023 at 12:14 PM Yubiao Feng > wrote: > > > > Hi Donglai, Mattison > > > > I agree with @Mattison > > > > Thanks > > Yubiao Feng > > > > On Mon, Aug 21, 2023 at 8:50 PM wrote: > > > > > > > > Hi, > > > > > > I agree with this change to improve the stability of the pulsar cluster. > > > > > > Just one concern. it's better to move this discussion to a new PIP. > > > because you wanna introduce a new broker configuration. > > > `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` > > > > > > FYI: https://github.com/apache/pulsar/blob/master/pip/README.md > > > > > > Looking forward this change and thanks for your contribution. :) > > > > > > Best, > > > Mattison > > > > > > > > > > > > On 7 Jul 2023 at 15:30 +0800, labuladong , wrote: > > > > Thanks you guys. > > > > > > > > > > > > I agree that per-minute is better than per-second, which is more > > > flexible. > > > > > > > > > > > > I open an issue here: > > > > > > > > > > > > https://github.com/apache/pulsar/issues/20753 > > > > > > > > > > > > Regards, > > > > donglai > > >
Re: [DISCUSS] Release Pulsar 3.1.1
+1 Zike Yang On Mon, Sep 25, 2023 at 2:55 PM Enrico Olivelli wrote: > > +1 > > Enrico > > Il giorno lun 25 set 2023 alle ore 05:08 Yubiao Feng > ha scritto: > > > > +1 > > > > Thanks > > Yubiao Feng > > > > On Mon, Sep 18, 2023 at 1:32 PM guo jiwei wrote: > > > > > Hi all, > > > > > > I would like to propose releasing the Pulsar 3.1.1. > > > > > > It's over one month since the release of 3.1.0 and there are 56 new > > > commits > > > in branch-3.1: > > > https://github.com/apache/pulsar/compare/v3.1.0...branch-3.1 > > > > > > We need to cut a new release. Please let me know if you have any > > > important fixes that need to be included in Pulsar 3.1.1. > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > >
Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView
Hi, Xiangying This PIP is a little confusing to me. This mail title states that we are introducing `readAllExistingMessages`. But actually, we only introduced `refreshAsync` in the PIP: https://github.com/apache/pulsar/pull/21166/files#diff-45c655583d6c0c73d87afd3df3fe67f77caadbf1bd691cf8f8211cc89728a1ceR34-R36 And the PR title doesn't seem relevant. “PIP-302 Add alwaysRefresh Configuration Option for TableView to Read Latest Values” BR, Zike Yang On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng wrote: > > Hi dev, >This thread is to start a vote for PIP-302 Add new API > readAllExistingMessages for TableView. > Discuss thread: > https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q > PIP: https://github.com/apache/pulsar/pull/21166 > > BR, > Xiangying
Re: [DISCUSS] Release Pulsar 3.0.2
+1 BR, Zike Yang On Sun, Sep 24, 2023 at 11:58 PM Dave Fisher wrote: > > Please use the correct name of the project - Apache Pulsar. > > Thanks, > Dave > > Sent from my iPhone > > > On Sep 24, 2023, at 7:08 AM, Yubiao Feng > > wrote: > > > > Hi all > > > > I would like to propose releasing the Pulsar 3.0.2 > > > > It's over one month since the release of 3.1.0, and there are 82 new > > commits. > > > > in branch-3.0: > > https://github.com/apache/pulsar/compare/v3.0.1...branch-3.0 > > > > We need to cut a new release. Please let me know if you have any > > important fixes that need to be included in Pulsar 3.0.2 > > > > Thanks > > Yubiao Feng >
Re: [VOTE] PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index
+1 (non-binding) BR, Zike Yang On Mon, Sep 18, 2023 at 9:35 AM mattison chao wrote: > > +1 (binding) > > Best, > Mattison > On 18 Sep 2023 at 09:19 +0800, PengHui Li , wrote: > > +1 (binding) > > > > Thanks, > > Penghui > > > > On Thu, Sep 14, 2023 at 11:23 AM Cong Zhao wrote: > > > > > Hi, all > > > > > > Since there are no other concerns in the discussion, I'm delighted to > > > start the voting process for the PIP-286. > > > > > > Here is the link to the PIP: https://github.com/apache/pulsar/pull/20867 > > > > > > Thanks, > > > Cong Zhao > > >
Re: [VOTE] Pulsar DotPulsar Release 3.0.0 Candidate 1
> We may publish an X.Y.Z-rcN version on NuGet as a staged one for > verification. And publish an X.Y.Z version on NuGet as a formal one when the vote is passed. +1. Installing the NuGet package from a local package file is complicated and not recommended. Publishing to the NuGet would make the release validation easier. Thanks, Zike Yang On Tue, Sep 12, 2023 at 9:30 PM tison wrote: > > Thanks for your participants! > > The release is moved to > https://dist.apache.org/repos/dist/release/pulsar/pulsar-dotpulsar-3.0.0. > > Here are several follow-ups for the next release: > > * NOTICE file - Fixed by https://github.com/apache/pulsar-dotpulsar/pull/174 > . > * Write a RELEASE guide - I'll summarize to actions I did in this release. > * Automaze releases - https://github.com/apache/pulsar-dotpulsar/issues/117 > * Binary release - @blankenstei...@apache.org > told me that releasing nuget package to SVN is meaningless. We may publish > an X.Y.Z-rcN version on NuGet as a staged one for verification. And publish > an X.Y.Z version on NuGet as a formal one when the vote is passed. > > Best, > tison. > > > mattison chao 于2023年9月11日周一 13:01写道： > > > +1 (binding) > > > > • Checked signature and checksums > > • Built the client from the source > > • Ran basic example. > > > > > > Best, > > Mattison > > On 8 Sep 2023 at 17:57 +0800, tison , wrote: > > > +1 (binding) > > > > > > I checked > > > > > > - Signature and checksums match > > > - Build the client from the source > > > - Run examples > > > > > > Best, > > > tison. > > > > > > > > > Yunze Xu 于2023年9月8日周五 01:15写道： > > > > > > > +1 (binding) > > > > > > > > - Verified signature and checksums > > > > - Build from source with dotnet 7.0.400 on Windows 11 > > > > - Run the example by adding the dotpulsar 3.0.0 dependency > > > > > > > > But I think there are some points to improve with the document. I'm > > > > new to the dotnet CLI though I have some experiences of C# a few years > > > > ago. I created the project via Visual Studio. However, when I tried to > > > > add the dependency via `dotnet add` command, I found it failed with a > > > > project template created by Visual Studio. Eventually I have to create > > > > and run a project via dotnet CLI: > > > > > > > > ``` > > > > cd project-dir > > > > dotnet new console > > > > dotnet add package DotPulsar --version 3.0.0 > > > > # Edit the Program.cs > > > > dotnet build > > > > dotnet run > > > > ``` > > > > > > > > It would be helpful to provide more guide, even links to existings > > > > tutorial from MSDN. > > > > > > > > Thanks, > > > > Yunze > > > > > > > > On Tue, Sep 5, 2023 at 3:51 PM Zike Yang wrote: > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > - Verified signature and checksums > > > > > > - Install the nupkg file > > > > > > - Build the client from the source > > > > > > - Run examples > > > > > > > > > > > > But I find that the dotpulsar 3.0.0 has already been published to > > the > > > > > > nuget.org: https://www.nuget.org/packages/DotPulsar/3.0.0 > > > > > > > > > > > > BR, > > > > > > Zike Yang > > > > > > > > > > > > On Sun, Sep 3, 2023 at 11:48 AM tison > > wrote: > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > This is the first release candidate for Apache DotPulsar, > > version > > > > 3.0.0. > > > > > > > > > > > > > > > > It fixes the following issues: > > > > > > > > > > https://github.com/apache/pulsar-dotpulsar/compare/2.11.1...3.0.0 > > > > > > > > > > > > > > > > Please download the source files and review this release > > candidate: > > > > > > > > - Download the source package, verify shasum and asc > > > > > > > > - Follow the README.md to build and run the DotPulsar. > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > majority > > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > > > Source files: > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-dotpulsar-3.0.0-candidate-1/ > > > > > > > > > > > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the > > release: > > > > > > > > https://downloads.apache.org/pulsar/KEYS > > > > > > > > > > > > > > > > The tag to be voted upon: > > > > > > > > v3.0.0 > > > > > > > > https://github.com/apache/pulsar-dotpulsar/releases/tag/3.0.0 > > > > > > > > > > > > > > > > Please review and vote on the release candidate #1 for the > > version > > > > 3.0.0, > > > > > > > > as follows: > > > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > > [ ] -1, Do not approve the release (please provide specific > > comments) > > > > > > > > > > > > > > > > Best, > > > > > > > > tison. > > > > > >
Re: [DISCUSS] PIP-286: Add findNewestPosition method in the TopicCompactionService API
+1 for me. Thanks, Zike Yang On Mon, Sep 11, 2023 at 11:13 AM Cong Zhao wrote: > > Hi, guys > > I noticed that the only globally ordered fields in the metadata are > 'publishTime' and 'entryIndex' (offset), so I split the previous method into > `findEntryByPublishTime` and `findEntryByEntryIndex`, > this will make method definitions more precise. > > The method returns the entry directly instead of RawEntryMetadata to avoid > introducing a new interface, it's important to note that the caller needs to > release it manually. > > I already updated the proposal, please help review it. > > Thanks, > Cong Zhao > > On 2023/07/25 04:12:09 Cong Zhao wrote: > > Hi Pulsar Community, I am writing to start the discussion on PIP 286 PR > > with PIP contents: https://github.com/apache/pulsar/pull/20867 Thanks, > > Cong Zhao > >
[VOTE] Pulsar Client Go Release 0.11.1 Candidate 1
Hi everyone, Please review and vote on the release candidate #1 for the version 0.11.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the first release candidate for Apache Pulsar Go client, version 0.11.1. It fixes the following issues: https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1 Pulsar Client Go's KEYS file contains PGP keys we used to sign this release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS Please download these packages and review this release candidate: - Review release notes: https://github.com/apache/pulsar-client-go/pull/1092 - Download the source package (verify shasum, and asc) and follow the README.md to build and run the pulsar-client-go. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source file: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/ The tag to be voted upon: v0.11.1-candidate-1 https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1 SHA-512 checksums: d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182 apache-pulsar-client-go-0.11.1-src.tar.gz
Re: [VOTE] PIP-277: Add `current` option in the Clusters list cmd
+1 (non-binding) BR, Zike Yang On Wed, Sep 6, 2023 at 11:47 PM tison wrote: > > +1 (binding) > > Trivial with low risk. > > Best, > tison. > > > mattison chao 于2023年9月6日周三 22:50写道： > > > +1 (binding) > > > > Best, > > Mattison > > On 6 Sep 2023 at 22:37 +0800, PengHui Li , wrote: > > > +1 (binding) > > > > > > The proposal just added a new option to show which cluster is the current > > > cluster accessed. It will not break any existing behavior. > > > > > > Regards, > > > Penghui > > > > > > On Wed, Sep 6, 2023 at 4:21 PM guo jiwei wrote: > > > > > > > Hi dev, > > > > This thread is to start a vote for PIP-277: Add `current` option in the > > > > Clusters list cmd > > > > > > > > Discuss thread : > > > > https://lists.apache.org/thread/800r6ld5wg7bttbywmk38m1qx12hs6nl > > > > PIP: https://github.com/apache/pulsar/pull/20614 > > > > > > > > > > > > > > > > Regards > > > > Jiwei Guo (Tboy) > > > > > >
Re: [VOTE] PIP-297: Support terminating Function & Connector with the fatal exception
Thank you all. Close this vote by 3 binding +1s: - Enrico - Penghui Li - Mattison And 1 non-binding +1: Baodi Shi Thanks, ZIke Yang On Tue, Sep 5, 2023 at 9:25 AM wrote: > > +1 (binding) > > Best, > Mattison > On 5 Sep 2023 at 09:07 +0800, PengHui Li , wrote: > > +1 (binding) > > > > Thanks, > > Penghui > > > > On Tue, Sep 5, 2023 at 8:40 AM Baodi Shi wrote: > > > > > +1(non-binding) > > > > > > > > > Thanks, > > > Baodi Shi > > > > > > > > > On Sep 5, 2023 at 05:23:38, Enrico Olivelli wrote: > > > > > > > > +1 (binding) > > > > > > > > > > Enrico > > > > > > > > > > Il giorno lun 4 set 2023 alle ore 12:32 Zike Yang ha > > > > > scritto: > > > > > > > > > > > > > > > Hi, all > > > > > > > > > > > > > > > Since there are no other concerns in the discussion, I'm delighted to > > > > > > > > > > start the voting process for the PIP-297. > > > > > > > > > > > > > > > Here is the link to the PIP: > > > > > https://github.com/apache/pulsar/pull/21079 > > > > > > > > > > > > > > > BR, > > > > > > > > > > Zike Yang > > > > > > > > > > > > >
Re: [VOTE] Pulsar DotPulsar Release 3.0.0 Candidate 1
+1 (non-binding) - Verified signature and checksums - Install the nupkg file - Build the client from the source - Run examples But I find that the dotpulsar 3.0.0 has already been published to the nuget.org: https://www.nuget.org/packages/DotPulsar/3.0.0 BR, Zike Yang On Sun, Sep 3, 2023 at 11:48 AM tison wrote: > > Hi everyone, > > This is the first release candidate for Apache DotPulsar, version 3.0.0. > > It fixes the following issues: > https://github.com/apache/pulsar-dotpulsar/compare/2.11.1...3.0.0 > > Please download the source files and review this release candidate: > - Download the source package, verify shasum and asc > - Follow the README.md to build and run the DotPulsar. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Source files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-dotpulsar-3.0.0-candidate-1/ > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://downloads.apache.org/pulsar/KEYS > > The tag to be voted upon: > v3.0.0 > https://github.com/apache/pulsar-dotpulsar/releases/tag/3.0.0 > > Please review and vote on the release candidate #1 for the version 3.0.0, > as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > Best, > tison.
[VOTE] PIP-297: Support terminating Function & Connector with the fatal exception
Hi, all Since there are no other concerns in the discussion, I'm delighted to start the voting process for the PIP-297. Here is the link to the PIP: https://github.com/apache/pulsar/pull/21079 BR, Zike Yang
Re: [DISCUSS] PIP-297: Support raising exceptions using Function & Connector context
Hi, Enrico Thanks for your review. I have replied under the PR. BR, Zike Yang On Mon, Aug 28, 2023 at 6:39 PM Enrico Olivelli wrote: > > Zike, > > Il giorno lun 28 ago 2023 alle ore 11:53 Zike Yang ha > scritto: > > > > Hi, all > > > > I opened a new PIP: https://github.com/apache/pulsar/pull/21079 > > > > This PIP is to improve the current function exception handler. It will > > be applied to both Pulsar Function and Pulsar IO Connector. > > > > Please feel free to give your feedback. Thanks! > > I have added feedback ont he GH issue. > > Summary: > - let's have two methods: raiseTemporaryError and raiseFatalError > - please clarify what happens in the method is called on a thread that > is not the main thread > > Enrico > > > > > BR, > > Zike Yang
[DISCUSS] PIP-297: Support raising exceptions using Function & Connector context
Hi, all I opened a new PIP: https://github.com/apache/pulsar/pull/21079 This PIP is to improve the current function exception handler. It will be applied to both Pulsar Function and Pulsar IO Connector. Please feel free to give your feedback. Thanks! BR, Zike Yang
Re: [DISCUSS] Release DotPulsar 3.0.0
+1 I am looking forward to this new release. Thanks, Zike Yang On Mon, Aug 28, 2023 at 9:15 AM PengHui Li wrote: > > +1 > > Thanks for driving the release. > > Penghui > > On Fri, Aug 25, 2023 at 9:48 PM David Jensen wrote: > > > Dear Apache PMC and Committers > > > > Me and Daniel Blankensteiner (blankensteiner) would like to announce we are > > soon ready to release DotPulsar 3.0.0. > > > > The release contains breaking changes, therefor we bump to a new major > > version. > > > > Changelog > > > > ### Added > > - Added partitioned topic support for the Consumer and Reader (was > > already implemented for the Producer) > > - MessageId now includes an extra field for the topic > > - A TryParse method is added to MessageId. Therefore, it is now > > possible to parse a string into a MessageId object > > - Support for `ProducerAccessMode` to prevent multiple producers on a > > single topic > > - A new `Fenced` state for producers which is a final state > > - The ability to explicitly set compression information on an outgoing > > message using `MessageMetadata` (for sending pre-compressed messages) > > > > ### Changed > > - The DelayedStateMonitor extension method now invokes onStateLeft > > when the initial state change is to a final state > > > > ### Fixed > > - Issue preventing readers from correctly going into the `Faulted` state > > - Calling `await Send(...)` on a producer did not correctly terminate > > with an exception when a send operation failed (e.g. because the > > producer faulted) > > - The 'Partition' in 'MessageId' will now be set to the correct > > partition when producing to partitioned topics > > - The OnStateChangeFrom extension method with delay functionality > > returned the inputted state but should return the current state > > - The DelayedStateMonitor extension method invoked onStateLeft with > > the inputted state but should have invoked it with the current state > > > > ### Deprecated > > - GetLastMessageId of the Consumer and Reader is deprecated, and soon > > to be removed. Please use GetLastMessageIds instead. > > > > Repo: https://github.com/apache/pulsar-dotpulsar > > > > Greetings > > David Jensen > >
Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue
> Hi Zike
You can see the processMessageChunk method of the ConsumerImpl.
Ah. That seems like a regression bug introduced by
https://github.com/apache/pulsar/pull/18511. I have pushed a PR to fix
it: https://github.com/apache/pulsar/pull/21070
For the behavior before Pulsar 3.0.0. The consumer should assemble the
message using 3,4,5.
Thanks for pointing this out.
BR,
Zike Yang
On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng wrote:
>
> >> Consumer receive:
> >1. SequenceID: 0, ChunkID: 0
> >2. SequenceID: 0, ChunkID: 1
> >3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> >chunk and recycle its `chunkedMsgCtx`.
> >4. SequenceID: 0, ChunkID: 1 // chunkedMsgCtx == null Release it.
> >5. SequenceID: 0, ChunkID: 2 // chunkedMsgCtx == null Release it.
> >
> >I think this case is wrong. For the current implementation, the
> >message 3,4,5 will be assembled as a original large message.
>
> Hi Zike
> You can see the processMessageChunk method of the ConsumerImpl.
>
> ```
>
> ChunkedMessageCtx chunkedMsgCtx =
> chunkedMessagesMap.get(msgMetadata.getUuid());
>
> if (msgMetadata.getChunkId() == 0 && chunkedMsgCtx == null) {
> //assemble a chunkedMsgCtx and put into
> pendingChunkedMessageUuidQueue and chunkedMessagesMap.
> }
>
> if (chunkedMsgCtx == null || chunkedMsgCtx.chunkedMsgBuffer == null
> || msgMetadata.getChunkId() !=
> (chunkedMsgCtx.lastChunkedMessageId + 1)) {
> if (chunkedMsgCtx != null) {
> if (chunkedMsgCtx.chunkedMsgBuffer != null) {
> ReferenceCountUtil.safeRelease(chunkedMsgCtx.chunkedMsgBuffer);
> }
> chunkedMsgCtx.recycle();
> }
> chunkedMessagesMap.remove(msgMetadata.getUuid());
> compressedPayload.release();
> increaseAvailablePermits(cnx);
> }
>
> ```
>
> On Sat, Aug 26, 2023 at 3:48 PM Zike Yang wrote:
> >
> > > Consumer receive:
> > 1. SequenceID: 0, ChunkID: 0
> > 2. SequenceID: 0, ChunkID: 1
> > 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this
> > chunk and recycle its `chunkedMsgCtx`.
> > 4. SequenceID: 0, ChunkID: 1 // chunkedMsgCtx == null Release it.
> > 5. SequenceID: 0, ChunkID: 2 // chunkedMsgCtx == null Release it.
> >
> > I think this case is wrong. For the current implementation, the
> > message 3,4,5 will be assembled as a original large message.
> >
> > HI, Heesung
> >
> >
> > > I think brokers probably need to track map to dedup
> >
> > I propose a simpler solution in this mail thread earlier, which
> > doesn't need to introduce map :
> >
> > > I think a simple better approach is to only check the deduplication
> > for the last chunk of the large message. The consumer only gets the
> > whole message after receiving the last chunk. We don't need to check
> > the deduplication for all previous chunks. Also by doing this we only
> > need bug fixes, we don't need to introduce a new PIP.
> >
> > Could you explain or show a case in what cases would lead to this
> > simpler solution not working?
> >
> > Thanks,
> > Zike Yang
> >
> > On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn
> > wrote:
> > >
> > > > In this case, the consumer only can receive m1.
> > >
> > > Regarding this comment, can you explain how the consumer only receives m1?
> > > Here, m1's and m2's uuid and msgId will be different(if we suffix with a
> > > chunkingSessionId), although the sequence id is the same.
> > >
> > > > If we throw an exception when users use the same sequence to send the
> > > message.
> > > Do You mean `If "producers" throw an exception when users use the same
> > > sequence to send the message.`.
> > > Again, when the producers restart, they lose the last sequence id sent.
> > >
> > >
> > > > If we do not throw an exception when users use the same sequence to
> > > send the message.
> > >
> > > For this logic, how do we handle if the producer suddenly resends the
> > > chunked message with a different chunking scheme(e.g. maxMessageSize) ?
> > > uuid=1, sid=0, cid=0
> > > uuid=1, sid=0, cid=1
> > > uuid=2, sid=0, cid=0
> > > uuid=2, sid=0, cid=1
> > >
> > > We could refine what to track and algo logic on the broker side more, but
> > > do we agree that the broker chunk dedup logic is needed?
> > >
> > > I will continue to think more next week. Have a nice weekend.
> > >
> > >
> > >
> > &g
Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue
> Consumer receive: 1. SequenceID: 0, ChunkID: 0 2. SequenceID: 0, ChunkID: 1 3. SequenceID: 0, ChunkID: 0 // chunk ID out of order. Release this chunk and recycle its `chunkedMsgCtx`. 4. SequenceID: 0, ChunkID: 1 // chunkedMsgCtx == null Release it. 5. SequenceID: 0, ChunkID: 2 // chunkedMsgCtx == null Release it. I think this case is wrong. For the current implementation, the message 3,4,5 will be assembled as a original large message. HI, Heesung > I think brokers probably need to track map to dedup I propose a simpler solution in this mail thread earlier, which doesn't need to introduce map : > I think a simple better approach is to only check the deduplication for the last chunk of the large message. The consumer only gets the whole message after receiving the last chunk. We don't need to check the deduplication for all previous chunks. Also by doing this we only need bug fixes, we don't need to introduce a new PIP. Could you explain or show a case in what cases would lead to this simpler solution not working? Thanks, Zike Yang On Sat, Aug 26, 2023 at 1:38 PM Heesung Sohn wrote: > > > In this case, the consumer only can receive m1. > > Regarding this comment, can you explain how the consumer only receives m1? > Here, m1's and m2's uuid and msgId will be different(if we suffix with a > chunkingSessionId), although the sequence id is the same. > > > If we throw an exception when users use the same sequence to send the > message. > Do You mean `If "producers" throw an exception when users use the same > sequence to send the message.`. > Again, when the producers restart, they lose the last sequence id sent. > > > > If we do not throw an exception when users use the same sequence to > send the message. > > For this logic, how do we handle if the producer suddenly resends the > chunked message with a different chunking scheme(e.g. maxMessageSize) ? > uuid=1, sid=0, cid=0 > uuid=1, sid=0, cid=1 > uuid=2, sid=0, cid=0 > uuid=2, sid=0, cid=1 > > We could refine what to track and algo logic on the broker side more, but > do we agree that the broker chunk dedup logic is needed? > > I will continue to think more next week. Have a nice weekend. > > > > > On Fri, Aug 25, 2023 at 9:14 PM Xiangying Meng wrote: > > > Hi Heesung, > > > > Maybe we only need to maintain the last chunk ID in a map. > > Map map1. > > And we already have a map maintaining the last sequence ID. > > Map map2. > > > > If we do not throw an exception when users use the same sequence to > > send the message. > > > > For any incoming msg, m : > > chunk ID = -1; > > If m is a chunk message: > > chunk ID = m.chunkID. > > If m.currentSeqid < LastSeqId, dedup. > > If m.currentSeqid > LastSeqId && m.chunk ID = 0, nodedup > > if chunk ID exists in the map. > >Update it and log an error. This means there is an > > incomplete chunk message. > > If chunk ID does not exist in the map. > >Put it on the map. > > If m.currentSeqid == LastSeqId, > >1. if m.chunk ID == -1 || m.chunk ID == 0. dedup. > >2. If 1 <= m.chunkID <= total chunk. > > 1. If chunk ID does not exist in the map. dedup. > > 2. If chunk ID exists in the map. dedup. Check the order > > of the chunkID to determine whether dedup; > >3. If m.chunkID == total chunk, persistent the chunk and > > remove the chunkID in the map. > > > > > > If we throw an exception when users use the same sequence to send the > > message. > > > > For any incoming msg, m : > > chunk ID = 0; > > If m is a chunk message: > > chunk ID = m.chunkID. > >If m.currentSeqid < LastSeqId, dedup. > >If m.currentSeqid == LastSeqId. > >If chunkID > 0, Check the last chunkID to determine whether to > > dedup. > > If chunkID == 1, put chunkID into the map if absent. > >IF chunkID = 0, dedup. > > > > BR, > > xiangying > > > > On Sat, Aug 26, 2023 at 11:53 AM Heesung Sohn > > wrote: > > > > > > However, what If the producer jvm gets restarted after the broker > > persists > > > the m1 (but before updating their sequence id in their persistence > > > storage), and the producer is trying to resend the same msg(so m2) with > > the > > > same sequence id after restarting? > > > > > > > > > > > > > > > > > > On Fri, Aug 25, 2023 at 8:22 PM Xiangying Meng > > wrote: > >
Re: [VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader
+1 (non-binding) Thanks, Zike Yang On Fri, Aug 25, 2023 at 2:52 PM Xiangying Meng wrote: > > Hi Pulsar Community, > > This is the vote thread for PIP 296: > https://github.com/apache/pulsar/pull/21052 > > This PIP will help to improve the flexibility of Reader usage. > > Thanks, > Xiangying
Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue
HI xiangying > The rewind operation is seen in the test log. That seems weird. Not sure if this rewind is related to the chunk consuming. > 1. SequenceID: 0, ChunkID: 0 2. SequenceID: 0, ChunkID: 1 3. SequenceID: 0, ChunkID: 1 4. SequenceID: 0, ChunkID: 2 Such four chunks cannot be processed correctly by the consumer. How would this happen to get two duplicated and consecutive ChunkID-1 messages? The producer should guarantee to retry the whole chunked messages instead of some parts of the chunks. > But chunks 1, 2, 3, and 4 are still persisted in the topic. I think it's OK to persist them all in the topic. Is there any issue with doing that? > There is another point. The resend of the chunk message has a bug that I shared with you, and you fixed in PR [0]. It will make this case happen in another way. If the user sets the sequence ID manually, the case could be reproduced. BR, Zike Yang On Thu, Aug 24, 2023 at 8:48 PM Xiangying Meng wrote: > > >IIUC, this may change the existing behavior and may introduce > >inconsistencies. > >Suppose that we have a large message with 3 chunks. But the producer > >crashes and resends the message after sending the chunk-1. It will > >send a total of 5 messages to the Pulsar topic: > > > >1. SequenceID: 0, ChunkID: 0 > >2. SequenceID: 0, ChunkID: 1 > >3. SequenceID: 0, ChunkID: 0 -> This message will be dropped > >4. SequenceID: 0, ChunkID: 1-> Will also be dropped > >5. SequenceID: 0, ChunkID: 2-> The last chunk of the message > > Hi Zike > There is another point. The resend of the chunk message has a bug that > I shared with you, and you fixed in PR [0]. It will make this case > happen in another way. > Sample description for the bug: > Because the chunk message uses the same message metadata, if the chunk > is not sent out immediately. Then, when resending, all chunks of the > same chunk message use the chunk ID of the last chunk. > In this case, It should happen as: > 1. SequenceID: 0, ChunkID: 0 (Put op1 into `pendingMessages` and send) > 2. SequenceID: 0, ChunkID: 1 (Put op2 into `pendingMessages` and send) > 3. SequenceID: 0, ChunkID: 2 -> (Put op3 into `pendingMessages`) > 4. SequenceID: 0, ChunkID: 2 -> (Resend op1) > 5. SequenceID: 0, ChunkID: 2 -> (Resend op2) > 6. SequenceID: 0, ChunkID: 2 -> (Send op3) > > > BR, > Xiangying > > [0] - https://github.com/apache/pulsar/pull/21048 > > On Thu, Aug 24, 2023 at 8:09 PM Xiangying Meng wrote: > > > > >> This solution also cannot solve the out-of-order messages inside the > > >>chunks. For example, the above five messages will still be persisted. > > >The consumer already handles this case. The above 5 messages will all > > >be persisted but the consumer will skip message 1 and 2. > > >For messages 3, 4, and 5. The producer can guarantee these chunks are in > > >order. > > > > The rewind operation is seen in the test log. Every time an incorrect > > chunk message is received, it will rewind, and the code has yet to be > > studied in depth. > > If it does not call rewind, then this case is considered a workable > > case. Let's look at another case. > > 1. SequenceID: 0, ChunkID: 0 > > 2. SequenceID: 0, ChunkID: 1 > > 3. SequenceID: 0, ChunkID: 1 > > 4. SequenceID: 0, ChunkID: 2 > > Such four chunks cannot be processed correctly by the consumer. > > > > In fact, this solution is my original idea. The PR I mentioned in the > > first email above uses a similar solution and modifies the logic on > > the consumer side. > > Also, as I mentioned in the first email, this solution can only solve > > the problem of end-to-end duplication. But chunks 1, 2, 3, and 4 are > > still persisted in the topic. > > > > On Thu, Aug 24, 2023 at 3:00 PM Zike Yang wrote: > > > > > > Hi Heesung, > > > > > > > I believe in this PIP "similar to the existing "sequence ID map", > > > to facilitate effective filtering" actually means tracking the last > > > chunkId(not all chunk ids) on the broker side. > > > > > > With this simple solution, I think we don't need to track the > > > (sequenceID, chunkID) on the broker side at all. The broker just needs > > > to apply the deduplication logic to the last chunk instead of all > > > previous chunks. This PIP actually could do that, but it will > > > introduce a new data format and compatibility issue. > > > > > > > This is still a behavior change(deduping chunk messages on the broker), > > > and I believe we need to discuss this addition as a PIP. > > &
Re: [DISCUSS] PIP-293: Add configuration for batching/chunking for retry letter and dead letter producers
Hi, Rin Thanks for your PIP! Chunking/batching is the feature for the producer. It would be confusing if we introduce these concepts to the DeadLetterPolicy. Regarding the issue mentioned in this PIP. I have created a PR to try to fix it: https://github.com/apache/pulsar/pull/21048 I'm trying to enable chunking and disable batching for retry and DLQ producers. PTAL. Thanks. BR, Zike Yang On Sat, Aug 19, 2023 at 5:28 AM Rin Davis wrote: > > Hi Pulsar Community, This is the discussion thread for PIP > https://github.com/apache/pulsar/pull/20959 > This PIP will allow users to specify that chunking/batching is enabled on > retry letter and dead letter producers associated with a consumer. > > Thank you, > Rin
Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue
Hi Heesung, > I believe in this PIP "similar to the existing "sequence ID map", to facilitate effective filtering" actually means tracking the last chunkId(not all chunk ids) on the broker side. With this simple solution, I think we don't need to track the (sequenceID, chunkID) on the broker side at all. The broker just needs to apply the deduplication logic to the last chunk instead of all previous chunks. This PIP actually could do that, but it will introduce a new data format and compatibility issue. > This is still a behavior change(deduping chunk messages on the broker), and I believe we need to discuss this addition as a PIP. Actually, we didn't specifically state the deduping chunk message behavior before. The chunked message should be equally applicable to the de-duplication logic as a regular message. Therefore, I think it should be considered as a bug fix. But if this FIX is worth discussing in depth. I have no objection to it being a new PIP. > I think brokers can track the last chunkMaxMessageSize for each producer. Using different chunkMaxMessageSize is just one of the aspects. In PIP-132 [0], we have included the message metadata size when checking maxMessageSize. The message metadata can be changed after splitting the chunks. We are still uncertain about the way the chunked message is split, even using the same ss chunkMaxMessageSize. > then the brokers can assume that the producer is resending the chunks from the beginning with a different scheme(restarted with a different chunkMaxMessageSize) and accept those new chunks from the beginning. Regarding this, it seems like we are implementing dynamic configuration for the chunkMaxMessageSize. I'm afraid that this would change the expected behavior and introduce more complexity to this configuration. [0] https://github.com/apache/pulsar/pull/14007 BR, Zike Yang On Thu, Aug 24, 2023 at 2:21 PM Zike Yang wrote: > > Hi, xiangying > > > it will find that the message > is out of order and rewind the cursor. Loop this operation, and > discard this message after it expires instead of assembling 3, 4, 5 > into a message. > > Could you point out where the implementation for this? From my > understanding, there should not be any rewind operation for the > chunking feature. You can check more detail here: > https://streamnative.io/blog/deep-dive-into-message-chunking-in-pulsar#how-message-chunking-is-implemented > > > This solution also cannot solve the out-of-order messages inside the > chunks. For example, the above five messages will still be persisted. > > The consumer already handles this case. The above 5 messages will all > be persisted but the consumer will skip message 1 and 2. > For messages 3, 4, and 5. The producer can guarantee these chunks are in > order. > > BR, > Zike Yang > > On Thu, Aug 24, 2023 at 11:48 AM Yubiao Feng > wrote: > > > > > 1. SequenceID: 0, ChunkID: 0 > > > 2. SequenceID: 0, ChunkID: 1 > > > 3. SequenceID: 0, ChunkID: 0 > > > 4. SequenceID: 0, ChunkID: 1 > > > 5. SequenceID: 0, ChunkID: 2 > > > For the existing behavior, the consumer assembles > > > messages 3,4,5 into > > > the original large message. But the changes brought > > > about by this PIP > > > will cause the consumer to use messages 1,2,5 for > > > assembly. There is > > > no guarantee that the producer will split the message > > > in the same way > > > twice before and after. For example, the producer's > > > maxMessageSize may > > > be different. This may cause the consumer to > > > receive a corrupt > > > message. > > > > Good point. > > > > > > Thanks > > Yubiao Feng > > > > On Wed, Aug 23, 2023 at 12:34 PM Zike Yang wrote: > > > > > Hi, xiangying, > > > > > > Thanks for your PIP. > > > > > > IIUC, this may change the existing behavior and may introduce > > > inconsistencies. > > > Suppose that we have a large message with 3 chunks. But the producer > > > crashes and resends the message after sending the chunk-1. It will > > > send a total of 5 messages to the Pulsar topic: > > > > > > 1. SequenceID: 0, ChunkID: 0 > > > 2. SequenceID: 0, ChunkID: 1 > > > 3. SequenceID: 0, ChunkID: 0 -> This message will be dropped > > > 4. SequenceID: 0, ChunkID: 1-> Will also be dropped > > > 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message > > > > > > For the existing behavior, the consumer assembles messages 3,4,5 into > > > the original large message. But the changes brought about by this PIP > > > will cause the co
Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue
Hi, xiangying > it will find that the message is out of order and rewind the cursor. Loop this operation, and discard this message after it expires instead of assembling 3, 4, 5 into a message. Could you point out where the implementation for this? From my understanding, there should not be any rewind operation for the chunking feature. You can check more detail here: https://streamnative.io/blog/deep-dive-into-message-chunking-in-pulsar#how-message-chunking-is-implemented > This solution also cannot solve the out-of-order messages inside the chunks. For example, the above five messages will still be persisted. The consumer already handles this case. The above 5 messages will all be persisted but the consumer will skip message 1 and 2. For messages 3, 4, and 5. The producer can guarantee these chunks are in order. BR, Zike Yang On Thu, Aug 24, 2023 at 11:48 AM Yubiao Feng wrote: > > > 1. SequenceID: 0, ChunkID: 0 > > 2. SequenceID: 0, ChunkID: 1 > > 3. SequenceID: 0, ChunkID: 0 > > 4. SequenceID: 0, ChunkID: 1 > > 5. SequenceID: 0, ChunkID: 2 > > For the existing behavior, the consumer assembles > > messages 3,4,5 into > > the original large message. But the changes brought > > about by this PIP > > will cause the consumer to use messages 1,2,5 for > > assembly. There is > > no guarantee that the producer will split the message > > in the same way > > twice before and after. For example, the producer's > > maxMessageSize may > > be different. This may cause the consumer to > > receive a corrupt > > message. > > Good point. > > > Thanks > Yubiao Feng > > On Wed, Aug 23, 2023 at 12:34 PM Zike Yang wrote: > > > Hi, xiangying, > > > > Thanks for your PIP. > > > > IIUC, this may change the existing behavior and may introduce > > inconsistencies. > > Suppose that we have a large message with 3 chunks. But the producer > > crashes and resends the message after sending the chunk-1. It will > > send a total of 5 messages to the Pulsar topic: > > > > 1. SequenceID: 0, ChunkID: 0 > > 2. SequenceID: 0, ChunkID: 1 > > 3. SequenceID: 0, ChunkID: 0 -> This message will be dropped > > 4. SequenceID: 0, ChunkID: 1-> Will also be dropped > > 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message > > > > For the existing behavior, the consumer assembles messages 3,4,5 into > > the original large message. But the changes brought about by this PIP > > will cause the consumer to use messages 1,2,5 for assembly. There is > > no guarantee that the producer will split the message in the same way > > twice before and after. For example, the producer's maxMessageSize may > > be different. This may cause the consumer to receive a corrupt > > message. > > > > Also, this PIP increases the complexity of handling chunks on the > > broker side. Brokers should, in general, treat the chunk as a normal > > message. > > > > I think a simple better approach is to only check the deduplication > > for the last chunk of the large message. The consumer only gets the > > whole message after receiving the last chunk. We don't need to check > > the deduplication for all previous chunks. Also by doing this we only > > need bug fixes, we don't need to introduce a new PIP. > > > > BR, > > Zike Yang > > > > On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng > > wrote: > > > > > > Dear Community, > > > > > > I hope this email finds you well. I'd like to address an important > > > issue related to Apache Pulsar and discuss a solution I've proposed on > > > GitHub. The problem pertains to the handling of Chunk Messages after > > > enabling deduplication. > > > > > > In the current version of Apache Pulsar, all chunks of a Chunk Message > > > share the same sequence ID. However, enabling the depublication > > > feature results in an inability to send Chunk Messages. To tackle this > > > problem, I've proposed a solution [1] that ensures messages are not > > > duplicated throughout end-to-end delivery. While this fix addresses > > > the duplication issue for end-to-end messages, there remains a > > > possibility of duplicate chunks within topics. > > > > > > To address this concern, I believe we should introduce a "Chunk ID > > > map" at the Broker level, similar to the existing "sequence ID map", > > > to facilitate effective filtering. However, implementing this has led > > > to a challenge: a producer requires storage for two Long values > > > s
Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue
Hi, xiangying, Thanks for your PIP. IIUC, this may change the existing behavior and may introduce inconsistencies. Suppose that we have a large message with 3 chunks. But the producer crashes and resends the message after sending the chunk-1. It will send a total of 5 messages to the Pulsar topic: 1. SequenceID: 0, ChunkID: 0 2. SequenceID: 0, ChunkID: 1 3. SequenceID: 0, ChunkID: 0 -> This message will be dropped 4. SequenceID: 0, ChunkID: 1-> Will also be dropped 5. SequenceID: 0, ChunkID: 2-> The last chunk of the message For the existing behavior, the consumer assembles messages 3,4,5 into the original large message. But the changes brought about by this PIP will cause the consumer to use messages 1,2,5 for assembly. There is no guarantee that the producer will split the message in the same way twice before and after. For example, the producer's maxMessageSize may be different. This may cause the consumer to receive a corrupt message. Also, this PIP increases the complexity of handling chunks on the broker side. Brokers should, in general, treat the chunk as a normal message. I think a simple better approach is to only check the deduplication for the last chunk of the large message. The consumer only gets the whole message after receiving the last chunk. We don't need to check the deduplication for all previous chunks. Also by doing this we only need bug fixes, we don't need to introduce a new PIP. BR, Zike Yang On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng wrote: > > Dear Community, > > I hope this email finds you well. I'd like to address an important > issue related to Apache Pulsar and discuss a solution I've proposed on > GitHub. The problem pertains to the handling of Chunk Messages after > enabling deduplication. > > In the current version of Apache Pulsar, all chunks of a Chunk Message > share the same sequence ID. However, enabling the depublication > feature results in an inability to send Chunk Messages. To tackle this > problem, I've proposed a solution [1] that ensures messages are not > duplicated throughout end-to-end delivery. While this fix addresses > the duplication issue for end-to-end messages, there remains a > possibility of duplicate chunks within topics. > > To address this concern, I believe we should introduce a "Chunk ID > map" at the Broker level, similar to the existing "sequence ID map", > to facilitate effective filtering. However, implementing this has led > to a challenge: a producer requires storage for two Long values > simultaneously (sequence ID and chunk ID). Because the snapshot of the > sequence ID map is stored through the properties of the cursor > (Map), so in order to satisfy the storage of two Longs > (sequence ID, chunk ID) corresponding to one producer, we hope to add > a mark DeleteProperties (Map) String, String>) to > replace the properties (Map) field. To resolve this, > I've proposed an alternative proposal [2] involving the introduction > of a "mark DeleteProperties" (Map) to replace the > current properties (Map) field. > > I'd appreciate it if you carefully review both PRs and share your > valuable feedback and insights. Thank you immensely for your time and > attention. I eagerly anticipate your valuable opinions and > recommendations. > > Warm regards, > Xiangying > > [1] https://github.com/apache/pulsar/pull/20948 > [2] https://github.com/apache/pulsar/pull/21027
Re: [ANNOUNCE] Apache Pulsar 3.1.0 released
I have re-pushed the docker image, and the image for the intel arch can work now. BR, Zike Yang On Mon, Aug 14, 2023 at 6:29 PM Zike Yang wrote: > > Congratulations! > > I just found an issue with the pulsar:3.1.0 docker image: > There is only one arch supported `linux/arm64` for the pulsar:3.1.0: > https://hub.docker.com/layers/apachepulsar/pulsar/3.1.0/images/sha256-307031b9bde3733344417a4d72702bc6727fad019056fe7922c50b6386eda46c?context=explore > We need to push the `amd64` one. > > BR, > Zike Yang > > On Mon, Aug 14, 2023 at 5:28 PM Enrico Olivelli wrote: > > > > Congratulations! > > > > Enrico > > > > Il Lun 14 Ago 2023, 11:16 guo jiwei ha scritto: > >> > >> The Apache Pulsar team is proud to announce Apache Pulsar version 3.1.0. > >> > >> Pulsar is a highly scalable, low latency messaging platform running on > >> commodity hardware. It provides simple pub-sub semantics over topics, > >> guaranteed at-least-once delivery of messages, automatic cursor management > >> for > >> subscribers, and cross-datacenter replication. > >> > >> For Pulsar release details and downloads, visit: > >> > >> https://pulsar.apache.org/download > >> > >> Release Notes are at: > >> https://pulsar.apache.org/release-notes > >> > >> We would like to thank the contributors that made the release possible. > >> > >> > >> Regards > >> Jiwei Guo (Tboy)
Re: [Discuss] Release Pulsar Python Client 3.3.0
+1 BR, Zike Yang On Tue, Aug 15, 2023 at 7:19 PM Baodi Shi wrote: > > Hi all, > > I would like to propose releasing the Pulsar Python Client 3.3.0. > > It has been over 2 months since the last release (3.2.0). There have > been some new features and bug fixes since then. It's time to release > a new version. > > >- https://github.com/apache/pulsar-client-python/milestone/4?closed=1 > > > Please let me know if you have any PRs that need to be included in 3.3.0. > > > Thanks, > Baodi Shi
