Re: [VOTE] Apache Pulsar 2.8.2 candidate 1
Lin Lin, -1 (binding) I have verified the packages, all good: - unit tests are passing (many flaky tests) - RAT passes - build passes (JDK11 on Ubuntu) - Pulsar standalone works (basic smoke tests) But I am not able to verify the signatures. Please check the commands you used to stage the release, probably you used the wrong key I have imported the key from https://dist.apache.org/repos/dist/dev/pulsar/KEYS This is the output... Enrico gpg --verify apache-pulsar-2.8.2-bin.tar.gz.asc gpg: assuming signed data in 'apache-pulsar-2.8.2-bin.tar.gz' gpg: Signature made Wed 1 Dec 04:55:01 2021 CET gpg:using RSA key 460CEE75D2A7049C94B93FA2249DF015663A88B9 gpg: BAD signature from "linlin " [unknown] gpg: assuming signed data in 'apache-pulsar-2.8.2-src.tar.gz' gpg: Signature made Wed 1 Dec 04:55:06 2021 CET gpg:using RSA key 460CEE75D2A7049C94B93FA2249DF015663A88B9 gpg: BAD signature from "linlin " [unknown] gpg: assuming signed data in 'apache-pulsar-offloaders-2.8.2-bin.tar.gz' gpg: Signature made Wed 1 Dec 04:55:03 2021 CET gpg:using RSA key 460CEE75D2A7049C94B93FA2249DF015663A88B9 gpg: BAD signature from "linlin " [unknown] Il giorno mar 14 dic 2021 alle ore 11:25 Massimiliano Mirelli < massimilianomirelli...@gmail.com> ha scritto: > Thank you for the rc! > > I also found the same problem mentioned by Nicolo, when running: > > sha512sum -c apache-pulsar-2.8.2-bin.tar.gz > > I get the error: > > sha512sum: WARNING: 1 computed checksum did NOT match > > Also, when verifying the GPG keys, as well, running: > > gpg --verify apache-pulsar-2.8.2-bin.tar.gz > > throws: > > gpg: BAD signature from "linlin " [unknown] > > On Tue, 14 Dec 2021 at 10:47, Nicolò Boschi wrote: > > > Thank you for driving the release! > > > > I found out a problem with the checksum of apache-pulsar-2.8.2-bin.tar.gz > > > > You wrote in the email the sha512 is f51e93d5[..]683 and actually it is > > correct (shasum -a 512 apache-pulsar-2.8.2-bin.tar.gz > > But this file ( > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/apache-pulsar-2.8.2-bin.tar.gz.sha512 > > ) > > reports another checksum > > > > Can you please check and confirm? > > > > > > Il giorno mar 14 dic 2021 alle ore 04:26 linlin ha > > scritto: > > > > > This is the first release candidate for Apache Pulsar, version 2.8.2. > > > > > > It fixes the following issues: > > > > > > > > > https://github.com/apache/pulsar/issues?q=label%3Acherry-picked%2Fbranch-2.8+is%3Aclosed+label%3Arelease%2F2.8.2 > > > > > > *** Please download, test and vote on this release. This vote will stay > > > open > > > for at least 72 hours *** > > > > > > Note that we are voting upon the source (tag), binaries are provided > for > > > convenience. > > > > > > Source and binary files: > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/ > > > > > > SHA-512 checksums: > > > > > > > > > f51e93d5caa7ea4ec2616e096ca75dd71bccb475632ee5ff35d713b8f5112689d17315a1cd9350dd8f8f0bdc2e059be5fb179b2b8b3b39aae77e466103294683 > > > apache-pulsar-2.8.2-bin.tar.gz > > > > > > > > > 8540641e76fb541f9dbfaff263946ed19a585266e5de011e78188d78ec4e1c828e8893eb2e783a1ebad866f5513efffd93396b7abd77c347f34ab689badf4fad > > > apache-pulsar-2.8.2-src.tar.gz > > > > > > > > > Maven staging repo: > > > > https://repository.apache.org/content/repositories/orgapachepulsar-1108/ > > > > > > The tag to be voted upon: > > > v2.8.2-candidate-1 > > > https://github.com/apache/pulsar/releases/tag/v2.8.2-candidate-1 > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > Please download the source package, and follow the README to build > > > and run the Pulsar standalone service. > > > > > > Lin Lin > > > > > > > > > -- > > Nicolò Boschi > > >
Log4Shell Vulnerability
We should provide updates to secur...@apache.org about our status regarding Log4Shell. Here is the current blog post about impacted projects: https://blogs.apache.org/security/entry/cve-2021-44228 Here is a blog post about Log4j exploits; https://blogs.apache.org/foundation/entry/apache-log4j-cves Sent from my iPhone
Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher
+1 Enrico Il Mer 15 Dic 2021, 05:27 Michael Marshall ha scritto: > +1 > > - Michael > > On Tue, Dec 14, 2021 at 8:59 PM Shivji Kumar Jha > wrote: > > > > +1 > > > > Regards, > > Shivji Kumar Jha > > http://www.shivjijha.com/ > > +91 8884075512 > > > > > > On Wed, 15 Dec 2021 at 06:38, Hang Chen wrote: > > > > > +1 > > > > > > Thanks, > > > Hang > > > > > > PengHui Li 于2021年12月15日周三 07:51写道： > > > > > > > > +1 > > > > > > > > Penghui > > > > > > > > On Wed, Dec 15, 2021 at 2:15 AM Matteo Merli > wrote: > > > > > > > > > Pasted below for quoting convenience. > > > > > > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > > > The consistent hashing implementation to uniformly assign keys to > > > consumers > > > > > in the context of a KeyShared subscription, was introduced in > > > > > https://github.com/apache/pulsar/pull/6791, which was released in > > > Pulsar > > > > > 2.6.0. > > > > > > > > > > While consistent hashing can use slightly more memory in certain > > > cases, it > > > > > is > > > > > more suitable as a general default implementation, as it leads to a > > > fairer > > > > > distribution of keys across consumers, and avoiding corner cases > that > > > > > depend > > > > > on the sequence of addition/removal of consumers. > > > > > > > > > > ## Proposed changes > > > > > > > > > > In 2.10 release, for the setting: > > > > > > > > > > ```properties > > > > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use > > > > > splitting ranges or > > > > > # consistent hashing to reassign keys to new consumers > > > > > subscriptionKeySharedUseConsistentHashing=false > > > > > ``` > > > > > > > > > > Change its default value to `true`. > > > > > > > > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users > will > > > still > > > > > be > > > > > able to use the old implementation. > > > > > > > > > > > > > > > > > > > > -- > > > > > Matteo Merli > > > > > > > > > > > > > >
Re: [DISCUSSION] PIP-120: Enable client memory limit by default
+1
Enrico
Il Mer 15 Dic 2021, 06:03 ZhangJian He ha scritto:
> +1
>
> Thanks
> ZhangJian He
>
> Neng Lu 于2021年12月15日周三 12:52写道：
>
> > +1 (non-binding)
> >
> > On 2021/12/14 19:20:02 Matteo Merli wrote:
> > > https://github.com/apache/pulsar/issues/13306
> > >
> > >
> > > Pasted below for quoting convenience.
> > >
> > >
> > >
> > >
> > > ## Motivation
> > >
> > > In Pulsar 2.8, we have introduced a setting to control the amount of
> > memory
> > > used by a client instance.
> > >
> > > ```java
> > > interface ClientBuilder {
> > > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > > }
> > > ```
> > >
> > > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit
> is
> > being
> > > enforced.
> > >
> > > I think it's a good time for 2.10 to enable this setting by default
> and,
> > > correspondingly, to disable by default the producer queue size limit.
> > >
> > > This will simplify a lot the configuration that a producer application
> > will
> > > have to come up with, when publishing with many topic/partitions or
> > > when messages
> > > are bigger than expected.
> > >
> > > ## Proposed changes
> > >
> > > In 2.10 release, for the `ClientBuilder`, change
> > > * `memoryLimit`: 0 -> 64 MB
> > >
> > > For the `ProducerBuilder`, changes
> > > * `maxPendingMessages`: 1000 -> 0
> > >
> > > 64MB is picked because it's a small enough memory size that will
> > guarantee
> > > a very high producer throughput, irrespective of the individual
> messages
> > size.
> > >
> > >
> > >
> > > --
> > > Matteo Merli
> > >
> > >
> >
>
Re: CheckStyle Rule Change
Il Mer 15 Dic 2021, 06:38 ZhangJian He ha scritto: > Hello everyone, I have added checkstyle on pulsar-metadata, and > UnusedImports check on test code. If your PR is involved in these two > situations. Please pull the master code. Thanks > Got it Thanks Enrico > > Thanks > ZhangJian He >
CheckStyle Rule Change
Hello everyone, I have added checkstyle on pulsar-metadata, and UnusedImports check on test code. If your PR is involved in these two situations. Please pull the master code. Thanks Thanks ZhangJian He
Re: [DISCUSSION] PIP-120: Enable client memory limit by default
+1
Thanks
ZhangJian He
Neng Lu 于2021年12月15日周三 12:52写道：
> +1 (non-binding)
>
> On 2021/12/14 19:20:02 Matteo Merli wrote:
> > https://github.com/apache/pulsar/issues/13306
> >
> >
> > Pasted below for quoting convenience.
> >
> >
> >
> >
> > ## Motivation
> >
> > In Pulsar 2.8, we have introduced a setting to control the amount of
> memory
> > used by a client instance.
> >
> > ```java
> > interface ClientBuilder {
> > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > }
> > ```
> >
> > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> being
> > enforced.
> >
> > I think it's a good time for 2.10 to enable this setting by default and,
> > correspondingly, to disable by default the producer queue size limit.
> >
> > This will simplify a lot the configuration that a producer application
> will
> > have to come up with, when publishing with many topic/partitions or
> > when messages
> > are bigger than expected.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the `ClientBuilder`, change
> > * `memoryLimit`: 0 -> 64 MB
> >
> > For the `ProducerBuilder`, changes
> > * `maxPendingMessages`: 1000 -> 0
> >
> > 64MB is picked because it's a small enough memory size that will
> guarantee
> > a very high producer throughput, irrespective of the individual messages
> size.
> >
> >
> >
> > --
> > Matteo Merli
> >
> >
>
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 (non-binding) On 2021/12/14 17:18:03 Matteo Merli wrote: > https://github.com/apache/pulsar/issues/13302 > > Copying here for quoting convenience > > > > > > ## Motivation > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, where > all the components are started within the context of a single JVM process. > > Users are using the standalone as a way to get quickly started with Pulsar or > in all the cases where it makes sense to have a single node deployment. > > Right now, the standalone is starting by default with many components, > several of > which are quite complex, since they are designed to be deployed in a > distributed > fashion. > > ## Goal > > Simplify the components of Pulsar standalone to achieve: > > 1. Reduce complexity > 2. Reduce startup time > 3. Reduce memory and CPU footprint of running standalone > > ## Proposed changes > > The proposal here is to change some of the default implementations that are > used for the Pulsar standalone. > > 1. **Metadata Store implementation** --> > Change from ZooKeeper to RocksDB > > 2. **Pulsar functions package backend** --> > Change from using DistributedLog to using local filesystem, storing the > jars directly in the data folder instead of uploading them into BK. > > 3. **Pulsar functions state store implementation** --> > Change the state store to be backed by a MetadataStore based backed, > with the RocksDB implementation. > > 4. **Table Service** --> > Do not start BK table service by default > > ## Compatibility considerations > > In order to avoid compatibility issues where users have existing Pulsar > standalone services that they want to upgrade without conflicts, we will > follow the principle of keeping the old defaults where there is existing > data on the disk. > > We will add a file, serving the purpose as a flag, in the `data/standalone` > directory, for example `new-2.10-defaults`. > > If the file is present, or if the data directory is completely missing, we > will adopt the new set of default configuration settings. > > If the file is not there, we will continue to use existing defaults and we > will > not break the upgrade operation. > > > > > > -- > Matteo Merli > >
Re: [DISCUSSION] PIP-120: Enable client memory limit by default
+1 (non-binding)
On 2021/12/14 19:20:02 Matteo Merli wrote:
> https://github.com/apache/pulsar/issues/13306
>
>
> Pasted below for quoting convenience.
>
>
>
>
> ## Motivation
>
> In Pulsar 2.8, we have introduced a setting to control the amount of memory
> used by a client instance.
>
> ```java
> interface ClientBuilder {
> ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> }
> ```
>
> By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is being
> enforced.
>
> I think it's a good time for 2.10 to enable this setting by default and,
> correspondingly, to disable by default the producer queue size limit.
>
> This will simplify a lot the configuration that a producer application will
> have to come up with, when publishing with many topic/partitions or
> when messages
> are bigger than expected.
>
> ## Proposed changes
>
> In 2.10 release, for the `ClientBuilder`, change
> * `memoryLimit`: 0 -> 64 MB
>
> For the `ProducerBuilder`, changes
> * `maxPendingMessages`: 1000 -> 0
>
> 64MB is picked because it's a small enough memory size that will guarantee
> a very high producer throughput, irrespective of the individual messages size.
>
>
>
> --
> Matteo Merli
>
>
Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher
+1 - Michael On Tue, Dec 14, 2021 at 8:59 PM Shivji Kumar Jha wrote: > > +1 > > Regards, > Shivji Kumar Jha > http://www.shivjijha.com/ > +91 8884075512 > > > On Wed, 15 Dec 2021 at 06:38, Hang Chen wrote: > > > +1 > > > > Thanks, > > Hang > > > > PengHui Li 于2021年12月15日周三 07:51写道： > > > > > > +1 > > > > > > Penghui > > > > > > On Wed, Dec 15, 2021 at 2:15 AM Matteo Merli wrote: > > > > > > > Pasted below for quoting convenience. > > > > > > > > > > > > > > > > > > > > ## Motivation > > > > > > > > The consistent hashing implementation to uniformly assign keys to > > consumers > > > > in the context of a KeyShared subscription, was introduced in > > > > https://github.com/apache/pulsar/pull/6791, which was released in > > Pulsar > > > > 2.6.0. > > > > > > > > While consistent hashing can use slightly more memory in certain > > cases, it > > > > is > > > > more suitable as a general default implementation, as it leads to a > > fairer > > > > distribution of keys across consumers, and avoiding corner cases that > > > > depend > > > > on the sequence of addition/removal of consumers. > > > > > > > > ## Proposed changes > > > > > > > > In 2.10 release, for the setting: > > > > > > > > ```properties > > > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use > > > > splitting ranges or > > > > # consistent hashing to reassign keys to new consumers > > > > subscriptionKeySharedUseConsistentHashing=false > > > > ``` > > > > > > > > Change its default value to `true`. > > > > > > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will > > still > > > > be > > > > able to use the old implementation. > > > > > > > > > > > > > > > > -- > > > > Matteo Merli > > > > > > > > > >
[GitHub] [pulsar-client-node] hrsakai merged pull request #187: Bump the master version to 1.6.0
hrsakai merged pull request #187: URL: https://github.com/apache/pulsar-client-node/pull/187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
PIP 116: Create Pulsar Writing Style Guide
Hi Pulsar enthusiasts, As you may notice, there are more and more documentation contributions nowadays, which is great! Similar to coding style guides, in the field of technical writing, a writing style guide is a must to improve users' experience and eliminate writers' workload. To improve the doc quality, I create our style guide — *Pulsar Writing Style Guide* [1] # Definition Pulsar Writing Style Guide contains a set of standards for writing and designing content. It helps maintain a consistent style, voice, and tone across Pulsar documentation. This guide is inspired by some existing style guides. It references *IBM Style Guide*, *Chicago Manual of Style*, and other books for more exhaustive guidance. # Benefits Pulsar Writing Style Guide can improve the content quality and bring many benefits to users and writers like: For users: - Make documentation easier to read - Reduce users’ cognitive load - Increase users' confidence in the content’s authority For information developers: - Resolve arguments and support our needs to create readable, usable, and minimalist information - Save time and trouble by providing a single reference for writing about common topics, features, and more - Help write documentation in a clearer way and keep a consistent tone of voice and style >> This guide is not a one-shot job, the current version is an initial draft. I'll make continuous efforts on that. I believe that the Pulsar community will welcome this addition to our resources and profit from the instructions. Feel free to comment, thanks. [1] https://docs.google.com/document/d/1lc5j4RtuLIzlEYCBo97AC8-U_3Erzs_lxpkDuseU0n4/edit#
Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher
+1 Regards, Shivji Kumar Jha http://www.shivjijha.com/ +91 8884075512 On Wed, 15 Dec 2021 at 06:38, Hang Chen wrote: > +1 > > Thanks, > Hang > > PengHui Li 于2021年12月15日周三 07:51写道： > > > > +1 > > > > Penghui > > > > On Wed, Dec 15, 2021 at 2:15 AM Matteo Merli wrote: > > > > > Pasted below for quoting convenience. > > > > > > > > > > > > > > > ## Motivation > > > > > > The consistent hashing implementation to uniformly assign keys to > consumers > > > in the context of a KeyShared subscription, was introduced in > > > https://github.com/apache/pulsar/pull/6791, which was released in > Pulsar > > > 2.6.0. > > > > > > While consistent hashing can use slightly more memory in certain > cases, it > > > is > > > more suitable as a general default implementation, as it leads to a > fairer > > > distribution of keys across consumers, and avoiding corner cases that > > > depend > > > on the sequence of addition/removal of consumers. > > > > > > ## Proposed changes > > > > > > In 2.10 release, for the setting: > > > > > > ```properties > > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use > > > splitting ranges or > > > # consistent hashing to reassign keys to new consumers > > > subscriptionKeySharedUseConsistentHashing=false > > > ``` > > > > > > Change its default value to `true`. > > > > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will > still > > > be > > > able to use the old implementation. > > > > > > > > > > > > -- > > > Matteo Merli > > > > > > >
Re: [ANNOUNCE] New Committer: Marvin Cai
Congrats Marvin! Best Regards, Hang Aaron Williams 于2021年12月15日周三 02:37写道： > > Just want to echo all of the other messages of congratulations. Thank you > for all of your hard work! > > Aaron > > On Tue, Dec 14, 2021 at 12:58 AM Enrico Olivelli > wrote: > > > Kudos !! > > > > Welcome aboard > > > > Enrico > > > > Il giorno mar 14 dic 2021 alle ore 09:12 guo jiwei > > ha scritto: > > > > > Congratulations Marvin! > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Tue, Dec 14, 2021 at 3:44 PM Yu wrote: > > > > > > > Congratulations Marvin! > > > > > > > > On Tue, Dec 14, 2021 at 8:42 AM Guangning E > > > wrote: > > > > > > > > > Configrats! > > > > > > > > > > Thanks, > > > > > Guangning > > > > > > > > > > Huanli Meng 于2021年12月14日周二 08:34写道： > > > > > > > > > > > Congratulations Marvin! > > > > > > > > > > > > BR//Huanli > > > > > > > > > > > > > On Dec 13, 2021, at 5:46 PM, linlin wrote: > > > > > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited > > > > Marvin > > > > > > Cai > > > > > > > https://github.com/MarvinCai to become a committer and we are > > > > pleased > > > > > to > > > > > > > announce that he has accepted. > > > > > > > > > > > > > > Marvin has joined the community for more than 1 year now and he > > is > > > > > > active in > > > > > > > the Pulsar community for more than 6 months. > > > > > > > > > > > > > > Welcome and Congratulations, Marvin! > > > > > > > > > > > > > > Please join us in congratulating and welcoming Marvin onboard! > > > > > > > > > > > > > > Best Regards, > > > > > > > Lin Lin on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSSION] PIP-120: Enable client memory limit by default
+1
Thanks,
Hang
PengHui Li 于2021年12月15日周三 07:50写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 15, 2021 at 3:20 AM Matteo Merli wrote:
>
> > https://github.com/apache/pulsar/issues/13306
> >
> >
> > Pasted below for quoting convenience.
> >
> >
> >
> >
> > ## Motivation
> >
> > In Pulsar 2.8, we have introduced a setting to control the amount of memory
> > used by a client instance.
> >
> > ```java
> > interface ClientBuilder {
> > ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> > }
> > ```
> >
> > By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> > being
> > enforced.
> >
> > I think it's a good time for 2.10 to enable this setting by default and,
> > correspondingly, to disable by default the producer queue size limit.
> >
> > This will simplify a lot the configuration that a producer application will
> > have to come up with, when publishing with many topic/partitions or
> > when messages
> > are bigger than expected.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the `ClientBuilder`, change
> > * `memoryLimit`: 0 -> 64 MB
> >
> > For the `ProducerBuilder`, changes
> > * `maxPendingMessages`: 1000 -> 0
> >
> > 64MB is picked because it's a small enough memory size that will guarantee
> > a very high producer throughput, irrespective of the individual messages
> > size.
> >
> >
> >
> > --
> > Matteo Merli
> >
> >
Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher
+1 Thanks, Hang PengHui Li 于2021年12月15日周三 07:51写道： > > +1 > > Penghui > > On Wed, Dec 15, 2021 at 2:15 AM Matteo Merli wrote: > > > Pasted below for quoting convenience. > > > > > > > > > > ## Motivation > > > > The consistent hashing implementation to uniformly assign keys to consumers > > in the context of a KeyShared subscription, was introduced in > > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar > > 2.6.0. > > > > While consistent hashing can use slightly more memory in certain cases, it > > is > > more suitable as a general default implementation, as it leads to a fairer > > distribution of keys across consumers, and avoiding corner cases that > > depend > > on the sequence of addition/removal of consumers. > > > > ## Proposed changes > > > > In 2.10 release, for the setting: > > > > ```properties > > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use > > splitting ranges or > > # consistent hashing to reassign keys to new consumers > > subscriptionKeySharedUseConsistentHashing=false > > ``` > > > > Change its default value to `true`. > > > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still > > be > > able to use the old implementation. > > > > > > > > -- > > Matteo Merli > > > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 Thanks, Hang PengHui Li 于2021年12月15日周三 07:52写道： > > +1 > > Penghui > > On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli wrote: > > > https://github.com/apache/pulsar/issues/13302 > > > > Copying here for quoting convenience > > > > > > > > > > > > ## Motivation > > > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > > where > > all the components are started within the context of a single JVM process. > > > > Users are using the standalone as a way to get quickly started with Pulsar > > or > > in all the cases where it makes sense to have a single node deployment. > > > > Right now, the standalone is starting by default with many components, > > several of > > which are quite complex, since they are designed to be deployed in a > > distributed > > fashion. > > > > ## Goal > > > > Simplify the components of Pulsar standalone to achieve: > > > > 1. Reduce complexity > > 2. Reduce startup time > > 3. Reduce memory and CPU footprint of running standalone > > > > ## Proposed changes > > > > The proposal here is to change some of the default implementations that are > > used for the Pulsar standalone. > > > > 1. **Metadata Store implementation** --> > > Change from ZooKeeper to RocksDB > > > > 2. **Pulsar functions package backend** --> > > Change from using DistributedLog to using local filesystem, storing > > the > > jars directly in the data folder instead of uploading them into BK. > > > > 3. **Pulsar functions state store implementation** --> > > Change the state store to be backed by a MetadataStore based backed, > > with the RocksDB implementation. > > > > 4. **Table Service** --> > > Do not start BK table service by default > > > > ## Compatibility considerations > > > > In order to avoid compatibility issues where users have existing Pulsar > > standalone services that they want to upgrade without conflicts, we will > > follow the principle of keeping the old defaults where there is existing > > data on the disk. > > > > We will add a file, serving the purpose as a flag, in the `data/standalone` > > directory, for example `new-2.10-defaults`. > > > > If the file is present, or if the data directory is completely missing, we > > will adopt the new set of default configuration settings. > > > > If the file is not there, we will continue to use existing defaults and we > > will > > not break the upgrade operation. > > > > > > > > > > > > -- > > Matteo Merli > > > >
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
+1 Thanks, Hang PengHui Li 于2021年12月15日周三 07:59写道： > > +1 > > Penghui > > On Wed, Dec 15, 2021 at 2:03 AM Matteo Merli wrote: > > > https://github.com/apache/pulsar/issues/13304 > > > > > > Pasted below for quoting convenience. > > > > --- > > > > > > ## Motivation > > > > After all the work done for PIP-45 that was already included in 2.8 and 2.9 > > releases, it enabled the concept of re-acquirable resource locks and leader > > election. > > > > Another important change was to avoid doing any deferrable metadata > > operation > > when we know that we are not currently connected to the metadata service. > > > > Finally, that enabled stabilization in 2.9 the configuration setting that > > allows > > brokers to continue operating in a safe mode when the session with > > ZooKeeper > > expires. > > > > The way it works is that, when we lose a ZooKeeper session, the data plane > > will > > continue to work undisturbed, relying on the BookKeeper fencing to avoid > > any > > inconsistencies. > > > > New topics are not able to get started, but existing topics will see no > > impact. > > > > The original intention for shutting down the brokers was to ensure that we > > would automatically go back to a consistent state, with respect to which > > resources are "owned" in ZooKeeper by a given broker. > > > > With the re-acquirable resource locks, that problem was solved and > > thoroughly > > tested to be robust. > > > > ## Proposed changes > > > > In 2.10 release, for the setting: > > > > ```properties > > # There are two policies to apply when a broker metadata session > > expires: session expired happens, "shutdown" or "reconnect". > > # With "shutdown", the broker will be restarted. > > # With "reconnect", the broker will keep serving the topics, while > > attempting to recreate a new session. > > zookeeperSessionExpiredPolicy=shutdown > > ``` > > > > Change its default value to `reconnect`. > > > > > > -- > > Matteo Merli > > > >
[GitHub] [pulsar-client-node] hrsakai commented on issue #62: Listening for messages
hrsakai commented on issue #62: URL: https://github.com/apache/pulsar-client-node/issues/62#issuecomment-994185872 listener has already been supported, so I closed this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai closed issue #62: Listening for messages
hrsakai closed issue #62: URL: https://github.com/apache/pulsar-client-node/issues/62 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai edited a comment on issue #98: Support getRedeliveryCount of Message
hrsakai edited a comment on issue #98: URL: https://github.com/apache/pulsar-client-node/issues/98#issuecomment-994170279 `1.2.0` has already been released, so I closed this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai commented on issue #98: Support getRedeliveryCount of Message
hrsakai commented on issue #98: URL: https://github.com/apache/pulsar-client-node/issues/98#issuecomment-994170279 1.2.0` has already been released, so I closed this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai closed issue #98: Support getRedeliveryCount of Message
hrsakai closed issue #98: URL: https://github.com/apache/pulsar-client-node/issues/98 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai closed issue #136: Support message encryption/decryption
hrsakai closed issue #136: URL: https://github.com/apache/pulsar-client-node/issues/136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-client-node] hrsakai commented on issue #136: Support message encryption/decryption
hrsakai commented on issue #136: URL: https://github.com/apache/pulsar-client-node/issues/136#issuecomment-994165495 Fixed by https://github.com/apache/pulsar-client-node/pull/138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
+1 Penghui On Wed, Dec 15, 2021 at 2:03 AM Matteo Merli wrote: > https://github.com/apache/pulsar/issues/13304 > > > Pasted below for quoting convenience. > > --- > > > ## Motivation > > After all the work done for PIP-45 that was already included in 2.8 and 2.9 > releases, it enabled the concept of re-acquirable resource locks and leader > election. > > Another important change was to avoid doing any deferrable metadata > operation > when we know that we are not currently connected to the metadata service. > > Finally, that enabled stabilization in 2.9 the configuration setting that > allows > brokers to continue operating in a safe mode when the session with > ZooKeeper > expires. > > The way it works is that, when we lose a ZooKeeper session, the data plane > will > continue to work undisturbed, relying on the BookKeeper fencing to avoid > any > inconsistencies. > > New topics are not able to get started, but existing topics will see no > impact. > > The original intention for shutting down the brokers was to ensure that we > would automatically go back to a consistent state, with respect to which > resources are "owned" in ZooKeeper by a given broker. > > With the re-acquirable resource locks, that problem was solved and > thoroughly > tested to be robust. > > ## Proposed changes > > In 2.10 release, for the setting: > > ```properties > # There are two policies to apply when a broker metadata session > expires: session expired happens, "shutdown" or "reconnect". > # With "shutdown", the broker will be restarted. > # With "reconnect", the broker will keep serving the topics, while > attempting to recreate a new session. > zookeeperSessionExpiredPolicy=shutdown > ``` > > Change its default value to `reconnect`. > > > -- > Matteo Merli > >
Re: [DISCUSSION] PIP-117: Change Pulsar standalone defaults
+1 Penghui On Wed, Dec 15, 2021 at 1:18 AM Matteo Merli wrote: > https://github.com/apache/pulsar/issues/13302 > > Copying here for quoting convenience > > > > > > ## Motivation > > Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, > where > all the components are started within the context of a single JVM process. > > Users are using the standalone as a way to get quickly started with Pulsar > or > in all the cases where it makes sense to have a single node deployment. > > Right now, the standalone is starting by default with many components, > several of > which are quite complex, since they are designed to be deployed in a > distributed > fashion. > > ## Goal > > Simplify the components of Pulsar standalone to achieve: > > 1. Reduce complexity > 2. Reduce startup time > 3. Reduce memory and CPU footprint of running standalone > > ## Proposed changes > > The proposal here is to change some of the default implementations that are > used for the Pulsar standalone. > > 1. **Metadata Store implementation** --> > Change from ZooKeeper to RocksDB > > 2. **Pulsar functions package backend** --> > Change from using DistributedLog to using local filesystem, storing > the > jars directly in the data folder instead of uploading them into BK. > > 3. **Pulsar functions state store implementation** --> > Change the state store to be backed by a MetadataStore based backed, > with the RocksDB implementation. > > 4. **Table Service** --> > Do not start BK table service by default > > ## Compatibility considerations > > In order to avoid compatibility issues where users have existing Pulsar > standalone services that they want to upgrade without conflicts, we will > follow the principle of keeping the old defaults where there is existing > data on the disk. > > We will add a file, serving the purpose as a flag, in the `data/standalone` > directory, for example `new-2.10-defaults`. > > If the file is present, or if the data directory is completely missing, we > will adopt the new set of default configuration settings. > > If the file is not there, we will continue to use existing defaults and we > will > not break the upgrade operation. > > > > > > -- > Matteo Merli > >
Re: [DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher
+1 Penghui On Wed, Dec 15, 2021 at 2:15 AM Matteo Merli wrote: > Pasted below for quoting convenience. > > > > > ## Motivation > > The consistent hashing implementation to uniformly assign keys to consumers > in the context of a KeyShared subscription, was introduced in > https://github.com/apache/pulsar/pull/6791, which was released in Pulsar > 2.6.0. > > While consistent hashing can use slightly more memory in certain cases, it > is > more suitable as a general default implementation, as it leads to a fairer > distribution of keys across consumers, and avoiding corner cases that > depend > on the sequence of addition/removal of consumers. > > ## Proposed changes > > In 2.10 release, for the setting: > > ```properties > # On KeyShared subscriptions, with default AUTO_SPLIT mode, use > splitting ranges or > # consistent hashing to reassign keys to new consumers > subscriptionKeySharedUseConsistentHashing=false > ``` > > Change its default value to `true`. > > The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still > be > able to use the old implementation. > > > > -- > Matteo Merli > >
Re: [DISCUSSION] PIP-120: Enable client memory limit by default
+1
Penghui
On Wed, Dec 15, 2021 at 3:20 AM Matteo Merli wrote:
> https://github.com/apache/pulsar/issues/13306
>
>
> Pasted below for quoting convenience.
>
>
>
>
> ## Motivation
>
> In Pulsar 2.8, we have introduced a setting to control the amount of memory
> used by a client instance.
>
> ```java
> interface ClientBuilder {
> ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
> }
> ```
>
> By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is
> being
> enforced.
>
> I think it's a good time for 2.10 to enable this setting by default and,
> correspondingly, to disable by default the producer queue size limit.
>
> This will simplify a lot the configuration that a producer application will
> have to come up with, when publishing with many topic/partitions or
> when messages
> are bigger than expected.
>
> ## Proposed changes
>
> In 2.10 release, for the `ClientBuilder`, change
> * `memoryLimit`: 0 -> 64 MB
>
> For the `ProducerBuilder`, changes
> * `maxPendingMessages`: 1000 -> 0
>
> 64MB is picked because it's a small enough memory size that will guarantee
> a very high producer throughput, irrespective of the individual messages
> size.
>
>
>
> --
> Matteo Merli
>
>
[DISCUSSION] PIP-120: Enable client memory limit by default
https://github.com/apache/pulsar/issues/13306
Pasted below for quoting convenience.
## Motivation
In Pulsar 2.8, we have introduced a setting to control the amount of memory
used by a client instance.
```java
interface ClientBuilder {
ClientBuilder memoryLimit(long memoryLimit, SizeUnit unit);
}
```
By default, in 2.8 and 2.9 this setting is set to 0, meaning no limit is being
enforced.
I think it's a good time for 2.10 to enable this setting by default and,
correspondingly, to disable by default the producer queue size limit.
This will simplify a lot the configuration that a producer application will
have to come up with, when publishing with many topic/partitions or
when messages
are bigger than expected.
## Proposed changes
In 2.10 release, for the `ClientBuilder`, change
* `memoryLimit`: 0 -> 64 MB
For the `ProducerBuilder`, changes
* `maxPendingMessages`: 1000 -> 0
64MB is picked because it's a small enough memory size that will guarantee
a very high producer throughput, irrespective of the individual messages size.
--
Matteo Merli
Re: [ANNOUNCE] New Committer: Marvin Cai
Just want to echo all of the other messages of congratulations. Thank you for all of your hard work! Aaron On Tue, Dec 14, 2021 at 12:58 AM Enrico Olivelli wrote: > Kudos !! > > Welcome aboard > > Enrico > > Il giorno mar 14 dic 2021 alle ore 09:12 guo jiwei > ha scritto: > > > Congratulations Marvin! > > > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Tue, Dec 14, 2021 at 3:44 PM Yu wrote: > > > > > Congratulations Marvin! > > > > > > On Tue, Dec 14, 2021 at 8:42 AM Guangning E > > wrote: > > > > > > > Configrats! > > > > > > > > Thanks, > > > > Guangning > > > > > > > > Huanli Meng 于2021年12月14日周二 08:34写道： > > > > > > > > > Congratulations Marvin! > > > > > > > > > > BR//Huanli > > > > > > > > > > > On Dec 13, 2021, at 5:46 PM, linlin wrote: > > > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited > > > Marvin > > > > > Cai > > > > > > https://github.com/MarvinCai to become a committer and we are > > > pleased > > > > to > > > > > > announce that he has accepted. > > > > > > > > > > > > Marvin has joined the community for more than 1 year now and he > is > > > > > active in > > > > > > the Pulsar community for more than 6 months. > > > > > > > > > > > > Welcome and Congratulations, Marvin! > > > > > > > > > > > > Please join us in congratulating and welcoming Marvin onboard! > > > > > > > > > > > > Best Regards, > > > > > > Lin Lin on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > > >
[DISCUSSION] PIP-119: Enable consistent hashing by default on KeyShared dispatcher
Pasted below for quoting convenience. ## Motivation The consistent hashing implementation to uniformly assign keys to consumers in the context of a KeyShared subscription, was introduced in https://github.com/apache/pulsar/pull/6791, which was released in Pulsar 2.6.0. While consistent hashing can use slightly more memory in certain cases, it is more suitable as a general default implementation, as it leads to a fairer distribution of keys across consumers, and avoiding corner cases that depend on the sequence of addition/removal of consumers. ## Proposed changes In 2.10 release, for the setting: ```properties # On KeyShared subscriptions, with default AUTO_SPLIT mode, use splitting ranges or # consistent hashing to reassign keys to new consumers subscriptionKeySharedUseConsistentHashing=false ``` Change its default value to `true`. The `AUTO_SPLIT` mode will not be removed nor deprecated. Users will still be able to use the old implementation. -- Matteo Merli
[DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
https://github.com/apache/pulsar/issues/13304 Pasted below for quoting convenience. --- ## Motivation After all the work done for PIP-45 that was already included in 2.8 and 2.9 releases, it enabled the concept of re-acquirable resource locks and leader election. Another important change was to avoid doing any deferrable metadata operation when we know that we are not currently connected to the metadata service. Finally, that enabled stabilization in 2.9 the configuration setting that allows brokers to continue operating in a safe mode when the session with ZooKeeper expires. The way it works is that, when we lose a ZooKeeper session, the data plane will continue to work undisturbed, relying on the BookKeeper fencing to avoid any inconsistencies. New topics are not able to get started, but existing topics will see no impact. The original intention for shutting down the brokers was to ensure that we would automatically go back to a consistent state, with respect to which resources are "owned" in ZooKeeper by a given broker. With the re-acquirable resource locks, that problem was solved and thoroughly tested to be robust. ## Proposed changes In 2.10 release, for the setting: ```properties # There are two policies to apply when a broker metadata session expires: session expired happens, "shutdown" or "reconnect". # With "shutdown", the broker will be restarted. # With "reconnect", the broker will keep serving the topics, while attempting to recreate a new session. zookeeperSessionExpiredPolicy=shutdown ``` Change its default value to `reconnect`. -- Matteo Merli
[DISCUSSION] PIP-117: Change Pulsar standalone defaults
https://github.com/apache/pulsar/issues/13302 Copying here for quoting convenience ## Motivation Pulsar standalone is the "Pulsar in a box" version of a Pulsar cluster, where all the components are started within the context of a single JVM process. Users are using the standalone as a way to get quickly started with Pulsar or in all the cases where it makes sense to have a single node deployment. Right now, the standalone is starting by default with many components, several of which are quite complex, since they are designed to be deployed in a distributed fashion. ## Goal Simplify the components of Pulsar standalone to achieve: 1. Reduce complexity 2. Reduce startup time 3. Reduce memory and CPU footprint of running standalone ## Proposed changes The proposal here is to change some of the default implementations that are used for the Pulsar standalone. 1. **Metadata Store implementation** --> Change from ZooKeeper to RocksDB 2. **Pulsar functions package backend** --> Change from using DistributedLog to using local filesystem, storing the jars directly in the data folder instead of uploading them into BK. 3. **Pulsar functions state store implementation** --> Change the state store to be backed by a MetadataStore based backed, with the RocksDB implementation. 4. **Table Service** --> Do not start BK table service by default ## Compatibility considerations In order to avoid compatibility issues where users have existing Pulsar standalone services that they want to upgrade without conflicts, we will follow the principle of keeping the old defaults where there is existing data on the disk. We will add a file, serving the purpose as a flag, in the `data/standalone` directory, for example `new-2.10-defaults`. If the file is present, or if the data directory is completely missing, we will adopt the new set of default configuration settings. If the file is not there, we will continue to use existing defaults and we will not break the upgrade operation. -- Matteo Merli
Re: [discuss] BacklogQuota param change
This regression was already there in 2.8, and 2.9.x is already delayed
by 3 months. We cannot keep waiting indefinitely without shipping the
release.
--
Matteo Merli
On Tue, Dec 14, 2021 at 12:38 AM Enrico Olivelli wrote:
>
> ZhangJian
>
> it looks like a regression,
> I will wait for a patch before cutting the first RC for 2.9.1
>
> thanks for sharing this problem
>
> Enrico
>
> Il giorno mar 14 dic 2021 alle ore 07:19 PengHui Li ha
> scritto:
>
> > Thanks, ZhangJian.
> >
> > Looking forward to your PR.
> >
> > Penghui
> >
> >
> > On Tue, Dec 14, 2021 at 1:40 PM ZhangJian He wrote:
> >
> > > After discussing it with Matteo. It’s prob not backward compatible.
> > >
> > > I will work on a fix.
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > > ZhangJian He 于2021年12月14日周二 12:57写道：
> > >
> > > > I found a change between pulsar2.7 and pulsar2.9
> > > >
> > > > command
> > > > ```
> > > > bin/pulsar admin namespaces get-backlog-quotas $tenant/$namespace
> > > > ```
> > > >
> > > > pulsar2.7 returns `limit=1024, policy=consumer_backlog_eviction`
> > > >
> > > > but pulsar 2.9 returns
> > > >
> > > > ```[root@23da8000c7c1 bin]# ./pulsar-admin namespaces
> > get-backlog-quotas
> > > > public/functions
> > > > "destination_storageBacklogQuotaImpl(limitSize=102400,
> > limitTime=-1,
> > > > policy=consumer_backlog_eviction)"
> > > > ```
> > > >
> > > > I found that the @JsonAlias("limit") annotation has been removed
> > > > on org.apache.pulsar.common.policies.data.BacklogQuota in
> > > > https://github.com/apache/pulsar/pull/10774.
> > > > My question is, Is that a expected change or compatible change? I
> > search
> > > > the 2.8.0 release notes, I didn't saw it.
> > > >
> > > > And If it involves work, I'm happy to help :)
> > > >
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> > >
> >
Re: PIP 112: Generate Release Notes Automatically
Spot on. This also reminds me that we can create custom issue forms by adding YAML form definition files [1], which is more user-friendly and easy to maintain. [1] https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms#about-yaml-syntax-for-issue-forms On Tue, Dec 14, 2021 at 2:25 AM Michael Marshall wrote: > +1 Yu, thank you for putting together this thorough document. This is > a great initiative. > > I think it might help to review and possibly update the PR template as > part of this PIP. For example, the current template does not prompt > authors whether the PR should be mentioned in release notes. Such a > prompt could help committers determine the right labels for a PR. > > Thanks, > Michael > > On Mon, Dec 13, 2021 at 4:56 AM Li Li wrote: > > > > +1 > > > > Good idea, I think I can be part of this PIP after I finished upgrading > pulsar website. > > > > Thanks, > > LiLi > > > > > On Dec 13, 2021, at 4:18 PM, Yu wrote: > > > > > > Hi Pulsarers, > > > > > > As we know[1], there are some issues in the current Pulsar release > notes > > > (RN), for example: > > > > > > - For Pulsar users > > > They cannot capture the highlights quickly since the RN is a raw dump > of > > > PRs. > > > > > > - For Pulsar release managers (RM) > > > They feel overwhelmed by the **manual** workload of generating RN > since it > > > is created based on git commit messages, while many people do not > provide > > > clear and meaningful info. > > > It’s time-consuming to clear up all info especially for a major release > > > with lots of PRs. > > > > > > If RN is regarded as an afterthought and finished as a last-minute > task, it > > > is likely not written well. > > > Instead of rushing, treating RN as a part of development not only > reduces > > > RM's workload and makes communication more coordinated, > > > but also allows more time for us to choose the most valuable highlights > > > shown to users. > > > Consequently, the process of the current workflow should be improved. > > > > > > Therefore, I propose the PIP 112: Generate Release Notes Automatically > [2] > > > and add some initial thoughts and research there. > > > It is only a draft but I would like to invite you to join us to bring > > > another major change to Pulsar. I believe this would bring many > benefits to > > > all of us, thanks! > > > > > > [1] https://lists.apache.org/thread/dl3jb9p3zvlc6ntlkpmxf1m8dw5dcd8z > > > [2] > > > > https://github.com/apache/pulsar/wiki/PIP-112%3A-Generate-Release-Notes-Automatically > > >
Re: [VOTE] Apache Pulsar 2.8.2 candidate 1
Thank you for the rc! I also found the same problem mentioned by Nicolo, when running: sha512sum -c apache-pulsar-2.8.2-bin.tar.gz I get the error: sha512sum: WARNING: 1 computed checksum did NOT match Also, when verifying the GPG keys, as well, running: gpg --verify apache-pulsar-2.8.2-bin.tar.gz throws: gpg: BAD signature from "linlin " [unknown] On Tue, 14 Dec 2021 at 10:47, Nicolò Boschi wrote: > Thank you for driving the release! > > I found out a problem with the checksum of apache-pulsar-2.8.2-bin.tar.gz > > You wrote in the email the sha512 is f51e93d5[..]683 and actually it is > correct (shasum -a 512 apache-pulsar-2.8.2-bin.tar.gz > But this file ( > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/apache-pulsar-2.8.2-bin.tar.gz.sha512 > ) > reports another checksum > > Can you please check and confirm? > > > Il giorno mar 14 dic 2021 alle ore 04:26 linlin ha > scritto: > > > This is the first release candidate for Apache Pulsar, version 2.8.2. > > > > It fixes the following issues: > > > > > https://github.com/apache/pulsar/issues?q=label%3Acherry-picked%2Fbranch-2.8+is%3Aclosed+label%3Arelease%2F2.8.2 > > > > *** Please download, test and vote on this release. This vote will stay > > open > > for at least 72 hours *** > > > > Note that we are voting upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/ > > > > SHA-512 checksums: > > > > > f51e93d5caa7ea4ec2616e096ca75dd71bccb475632ee5ff35d713b8f5112689d17315a1cd9350dd8f8f0bdc2e059be5fb179b2b8b3b39aae77e466103294683 > > apache-pulsar-2.8.2-bin.tar.gz > > > > > 8540641e76fb541f9dbfaff263946ed19a585266e5de011e78188d78ec4e1c828e8893eb2e783a1ebad866f5513efffd93396b7abd77c347f34ab689badf4fad > > apache-pulsar-2.8.2-src.tar.gz > > > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1108/ > > > > The tag to be voted upon: > > v2.8.2-candidate-1 > > https://github.com/apache/pulsar/releases/tag/v2.8.2-candidate-1 > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > Please download the source package, and follow the README to build > > and run the Pulsar standalone service. > > > > Lin Lin > > > > > -- > Nicolò Boschi >
Re: [PIP-78] Reduce redundant producers from partitioned producer
sorry for the late reply, I missed this message. Il giorno mar 7 dic 2021 alle ore 05:56 Yuri Mizushima < yumiz...@yahoo-corp.jp> ha scritto: > Enrico, > Thank you for your comment. > > > IIUC with this change the client will control which metrics are reported > by > > the broker ? > > From the protocol perspective, yes. > However, the main point of this change is not to "control" metrics by the > client side, > but to make the broker aggregate partitioned topic's producer metrics > explicitly. > > Do you suggest adding a broker config that > configures whether partitioned producer stats are aggregated by > producerName instead of > introducing a backward compatibility key (i.e., > partial_producer_supported) on the client-side? > Not exactly. I would add a configuration parameter to allow the client to use this feature. So: - feature enabled -> everything works like in your patch - feature disabled -> the broker ignores *partial_producer_supported *and runs as before This way if you want to let the client aggregate the stats you can do it but by default clients are not able to interact with the metrics format. In multi-tenancy environments, like in large clusters shared by many teams, you do not want that the client is able to change the way the broker reports its metrics or in general that the client is able to alter the monitoring systems. Enrico > > It is simple. However, I think we can't enable the config until all > clients are updated. > > What do you think? > Regards, > -- > Yuri Mizushima > yumiz...@yahoo-corp.jp > > > On 2021/12/02 17:37, "Enrico Olivelli" wrote: > > Yuri, > IIUC with this change the client will control which metrics are > reported by > the broker ? > > I am not sure it is a good idea, because metrics are usually managed > by the > owners of the brokers, who sometimes are not the same who run the > clients. > > Also, I am not sure if this way it is possible for the client to force > the > Broker to create many metrics and create some kind of damage. > > Would it be better to add a Broker configuration flag to turn on this > feature ? I mean to allow the client to select the type of metrics ? > > > Enrico > > > Il giorno gio 2 dic 2021 alle ore 03:00 Yuri Mizushima < > yumiz...@yahoo-corp.jp> ha scritto: > > > Do you have any comments? > > If there are no comments by Dec. 7, I will close the discussion and > rebase > > the PR commit to current master. > > > > Regards, > > > > -- > > Yuri Mizushima > > yumiz...@yahoo-corp.jp > > > > > > On 2021/11/16 15:46, "Yuri Mizushima" > wrote: > > > > Dear Pulsar community, > > > > I have created a new PR > > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=JVOJsF5wTRoAyvy%2F8kyddBJP1XsdBv3YcIt2FP4OkHI%3Dreserved=0 > > for stats aggregation, > > but I didn't discuss about the wire protocol change. I hope we > will > > discuss it here. > > > > Currently, partitioned producer can't aggregate by any key such > as > > cnx, producerId, producerName, and so on. > > I think we need to add any aggregation system. > > Therefore, added new aggregation policy as producerName (with > client > > side implementation). > > > > New protocol field partial_producer_supported is not used for > stats > > aggregation. It is used for backward compatibility. > > > > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fpull%2F12401%2Ffiles%23diff-f29399fed32e0916cf28452ba71078a3ae5ed77bbaef9f52a925165d8ee66cfdR489data=04%7C01%7Cyumizush%40yahoo-corp.jp%7Ca56c759c260a46d176b408d9b56eb442%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637740310655072887%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=L%2B4JftwNnN7dUtck8wjWhwvqKhyFpyUcOcgsoinwyfQ%3Dreserved=0 > > > > In my understanding, if introduce new stats aggregation key to > client > > side, > > need a way to determine whether the feature is enabled at client > side. > > For example, whether the producer has specific field or metadata, > > the version (e.g. protocol version) is greater than threshold, > etc. > > > > Of course, if we can introduce aggregation feature without > adding any > > new key or implementations from client side, > > we can support the feature not only new client but also old one. > > > > I'm looking forward to your opinions or suggestions to this PR. > > > > Regards, > > -- > > Yuri
Re: [ANNOUNCE] New Committer: Marvin Cai
Kudos !! Welcome aboard Enrico Il giorno mar 14 dic 2021 alle ore 09:12 guo jiwei ha scritto: > Congratulations Marvin! > > > Regards > Jiwei Guo (Tboy) > > > On Tue, Dec 14, 2021 at 3:44 PM Yu wrote: > > > Congratulations Marvin! > > > > On Tue, Dec 14, 2021 at 8:42 AM Guangning E > wrote: > > > > > Configrats! > > > > > > Thanks, > > > Guangning > > > > > > Huanli Meng 于2021年12月14日周二 08:34写道： > > > > > > > Congratulations Marvin! > > > > > > > > BR//Huanli > > > > > > > > > On Dec 13, 2021, at 5:46 PM, linlin wrote: > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited > > Marvin > > > > Cai > > > > > https://github.com/MarvinCai to become a committer and we are > > pleased > > > to > > > > > announce that he has accepted. > > > > > > > > > > Marvin has joined the community for more than 1 year now and he is > > > > active in > > > > > the Pulsar community for more than 6 months. > > > > > > > > > > Welcome and Congratulations, Marvin! > > > > > > > > > > Please join us in congratulating and welcoming Marvin onboard! > > > > > > > > > > Best Regards, > > > > > Lin Lin on behalf of the Pulsar PMC > > > > > > > > > > > > > >
[VOTE] Pulsar Node.js Client Release 1.5.0 Candidate 1
Hi everyone, Please review and vote on the release candidate #1 for the version 1.5.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This is the first release candidate for Apache Pulsar Node.js client, version 1.5.0. It fixes the following issues: https://github.com/apache/pulsar-client-node/milestone/7?closed=1 Please download the source files and review this release candidate: - Review release notes - Download the source package (verify shasum and asc) and follow the README.md to build and run the Pulsar Node.js client. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Source files: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.5.0-candidate-1/ Pulsar's KEYS file containing PGP keys we use to sign the release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS SHA-512 checksum: 5c05c5368fb822165dde5c3aa758d600eca4452e1cc160f9b53311e27436e4cd52fbc55ca6382bb56019e4b0a3a6a409f73eec7c19844fb60715e41348fd8c09 pulsar-client-node-1.5.0.tar.gz The tag to be voted upon: v1.5.0-rc.1 https://github.com/apache/pulsar-client-node/releases/tag/v1.5.0-rc.1 Masahiro Sakamoto Yahoo Japan Corp. E-mail: massa...@yahoo-corp.jp
Re: [VOTE] Apache Pulsar 2.8.2 candidate 1
Thank you for driving the release! I found out a problem with the checksum of apache-pulsar-2.8.2-bin.tar.gz You wrote in the email the sha512 is f51e93d5[..]683 and actually it is correct (shasum -a 512 apache-pulsar-2.8.2-bin.tar.gz But this file ( https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/apache-pulsar-2.8.2-bin.tar.gz.sha512) reports another checksum Can you please check and confirm? Il giorno mar 14 dic 2021 alle ore 04:26 linlin ha scritto: > This is the first release candidate for Apache Pulsar, version 2.8.2. > > It fixes the following issues: > > https://github.com/apache/pulsar/issues?q=label%3Acherry-picked%2Fbranch-2.8+is%3Aclosed+label%3Arelease%2F2.8.2 > > *** Please download, test and vote on this release. This vote will stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.2-candidate-1/ > > SHA-512 checksums: > > f51e93d5caa7ea4ec2616e096ca75dd71bccb475632ee5ff35d713b8f5112689d17315a1cd9350dd8f8f0bdc2e059be5fb179b2b8b3b39aae77e466103294683 > apache-pulsar-2.8.2-bin.tar.gz > > 8540641e76fb541f9dbfaff263946ed19a585266e5de011e78188d78ec4e1c828e8893eb2e783a1ebad866f5513efffd93396b7abd77c347f34ab689badf4fad > apache-pulsar-2.8.2-src.tar.gz > > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1108/ > > The tag to be voted upon: > v2.8.2-candidate-1 > https://github.com/apache/pulsar/releases/tag/v2.8.2-candidate-1 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download the source package, and follow the README to build > and run the Pulsar standalone service. > > Lin Lin > -- Nicolò Boschi
Re: [discuss] BacklogQuota param change
ZhangJian
it looks like a regression,
I will wait for a patch before cutting the first RC for 2.9.1
thanks for sharing this problem
Enrico
Il giorno mar 14 dic 2021 alle ore 07:19 PengHui Li ha
scritto:
> Thanks, ZhangJian.
>
> Looking forward to your PR.
>
> Penghui
>
>
> On Tue, Dec 14, 2021 at 1:40 PM ZhangJian He wrote:
>
> > After discussing it with Matteo. It’s prob not backward compatible.
> >
> > I will work on a fix.
> >
> > Thanks
> > ZhangJian He
> >
> > ZhangJian He 于2021年12月14日周二 12:57写道：
> >
> > > I found a change between pulsar2.7 and pulsar2.9
> > >
> > > command
> > > ```
> > > bin/pulsar admin namespaces get-backlog-quotas $tenant/$namespace
> > > ```
> > >
> > > pulsar2.7 returns `limit=1024, policy=consumer_backlog_eviction`
> > >
> > > but pulsar 2.9 returns
> > >
> > > ```[root@23da8000c7c1 bin]# ./pulsar-admin namespaces
> get-backlog-quotas
> > > public/functions
> > > "destination_storageBacklogQuotaImpl(limitSize=102400,
> limitTime=-1,
> > > policy=consumer_backlog_eviction)"
> > > ```
> > >
> > > I found that the @JsonAlias("limit") annotation has been removed
> > > on org.apache.pulsar.common.policies.data.BacklogQuota in
> > > https://github.com/apache/pulsar/pull/10774.
> > > My question is, Is that a expected change or compatible change? I
> search
> > > the 2.8.0 release notes, I didn't saw it.
> > >
> > > And If it involves work, I'm happy to help :)
> > >
> > >
> > > Thanks
> > > ZhangJian He
> > >
> >
>
Re: [ANNOUNCE] New Committer: Marvin Cai
Congratulations Marvin! Regards Jiwei Guo (Tboy) On Tue, Dec 14, 2021 at 3:44 PM Yu wrote: > Congratulations Marvin! > > On Tue, Dec 14, 2021 at 8:42 AM Guangning E wrote: > > > Configrats! > > > > Thanks, > > Guangning > > > > Huanli Meng 于2021年12月14日周二 08:34写道： > > > > > Congratulations Marvin! > > > > > > BR//Huanli > > > > > > > On Dec 13, 2021, at 5:46 PM, linlin wrote: > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited > Marvin > > > Cai > > > > https://github.com/MarvinCai to become a committer and we are > pleased > > to > > > > announce that he has accepted. > > > > > > > > Marvin has joined the community for more than 1 year now and he is > > > active in > > > > the Pulsar community for more than 6 months. > > > > > > > > Welcome and Congratulations, Marvin! > > > > > > > > Please join us in congratulating and welcoming Marvin onboard! > > > > > > > > Best Regards, > > > > Lin Lin on behalf of the Pulsar PMC > > > > > > > > >
