Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
+0 (binding) - Checked the signature - Build from source (Java 17, Ubuntu 20.04 WSL2) - Start standalone with KoP branch-2.11 (ab9d761f) - Verify Pulsar client 2.11.0, Kafka client 3.3.1 (through KoP) - Verify pulsar-shell on Ubuntu 20.04 WSL2 and Windows: 1. Use `client` command to produce and consume messages 2. Use `admin` command to create, list and delete topics I didn't give +1 because I have to modify the `bin/pulsar-admin-common.cmd` to make `pulsar-shell` work on Windows. I left a command here: https://github.com/apache/pulsar/pull/17243#discussion_r1061199501. Not sure if it's a blocker so I didn't give +1 or -1 until someone could explain it. Thanks, Yunze On Wed, Jan 4, 2023 at 2:37 PM PengHui Li wrote: > > > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz and > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > I noticed here is the discussion about this part in Slack > https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1672411025574369 > > Penghui > > On Wed, Jan 4, 2023 at 2:35 PM Yunze Xu > wrote: > > > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz and > > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > > > Thanks, > > Yunze > > > > On Mon, Jan 2, 2023 at 8:01 PM guo jiwei wrote: > > > > > > This is the fifth release candidate for Apache Pulsar, version 2.11.0. > > > > > > This release contains 1617 commits by 61 contributors. > > > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-5 > > > > > > CI for this release candidate > > > https://github.com/Technoboy-/pulsar/pull/26 > > > > > > *** Please download, test and vote on this release. This vote will stay > > open > > > for at least 72 hours *** > > > > > > Note that we are voting upon the source (tag), binaries are provided for > > > convenience. > > > > > > Source and binary files: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-5 > > > > > > SHA-512 checksums: > > > > > > > > 5bf77f600ac23c7ec72696655a49801e0755d67fb4c59c7ab30e9449b061aacef61ca227f106e6b7f5f490fc2ac57412ac2af3ff064ba1e6a31c162d4ca2ba6d > > > > > > ./apache-pulsar-2.11.0-bin.tar.gz > > > > > > > > 0a3cab04b4a95ced3745d8b41fc3fa2632c91f77cb10e5c7e008b09ba2f279fc14de109626a44a9482b5354c5fa17bbc07ee369a32a252e358298e494c0fc1cb > > > > > > ./apache-pulsar-2.11.0-src.tar.gz > > > > > > Maven staging repo: > > > https://repository.apache.org/content/repositories/orgapachepulsar-1203/ > > > > > > The tag to be voted upon: > > > v2.11.0-candidate-5 (97ee1a114aa1e5df05d86a9dd47f7fe7a2194211) > > > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-5 > > > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > > > > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc5/images/sha256-548a518bb18890ddf8e48f0bad62695cfc4f743b9e1223ca45b2e03dba6e9149 > > > > > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc5/images/sha256-3b755fb67d49abeb7ab6a76b7123cc474375e3881526db26f43c8cfccdaa3cf6 > > > > > > Please download the source package, and follow the > > > release-candidate-validation doc to build > > > and run the Pulsar standalone service. > > > https://pulsar.apache.org/contribute/validate-release-candidate > > > > > > Since the metadata store is changed from ZK to RocksDB, the verification > > of > > > the `stateful functions` needs to set the parameter "export > > > PULSAR_STANDALONE_USE_ZOOKEEPER=1" > > > > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > >
Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar
Congrats! - Lan Liang Replied Message | From | Michael Marshall | | Date | 1/4/2023 13:09 | | To | | | Subject | Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar | Congratulations, Yunze! - Michael On Tue, Jan 3, 2023 at 8:26 PM houxiaoyu wrote: Congrats! Best, Hou xiaoyu Qiang Huang 于2023年1月3日周二 14:53写道： Congratulations!!! Yunze Jun Ma 于2022年12月31日周六 17:26写道： Congrats and well deserved! From: Dave Fisher Sent: Saturday, December 31, 2022 1:52 To: dev@pulsar.apache.org Subject: Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar Congratulations Yunze! Well deserved! All the best, Dave On Dec 29, 2022, at 4:42 AM, Haiting Jiang wrote: Hi all, The Apache Pulsar Project Management Committee (PMC) has invited Yunze Xu (https://github.com/BewareMyPower) as a member of the PMC and we are pleased to announce that he has accepted. He is very active in the community in the past few years and made a lot of great contributions. Welcome Yunze to the Apache Pulsar PMC. Best Regards, Haiting Jiang on behalf of the Pulsar PMC -- BR, Qiang Huang
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
> I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz and apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? I noticed here is the discussion about this part in Slack https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1672411025574369 Penghui On Wed, Jan 4, 2023 at 2:35 PM Yunze Xu wrote: > I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz and > apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? > > Thanks, > Yunze > > On Mon, Jan 2, 2023 at 8:01 PM guo jiwei wrote: > > > > This is the fifth release candidate for Apache Pulsar, version 2.11.0. > > > > This release contains 1617 commits by 61 contributors. > > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-5 > > > > CI for this release candidate > > https://github.com/Technoboy-/pulsar/pull/26 > > > > *** Please download, test and vote on this release. This vote will stay > open > > for at least 72 hours *** > > > > Note that we are voting upon the source (tag), binaries are provided for > > convenience. > > > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-5 > > > > SHA-512 checksums: > > > > > 5bf77f600ac23c7ec72696655a49801e0755d67fb4c59c7ab30e9449b061aacef61ca227f106e6b7f5f490fc2ac57412ac2af3ff064ba1e6a31c162d4ca2ba6d > > > > ./apache-pulsar-2.11.0-bin.tar.gz > > > > > 0a3cab04b4a95ced3745d8b41fc3fa2632c91f77cb10e5c7e008b09ba2f279fc14de109626a44a9482b5354c5fa17bbc07ee369a32a252e358298e494c0fc1cb > > > > ./apache-pulsar-2.11.0-src.tar.gz > > > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapachepulsar-1203/ > > > > The tag to be voted upon: > > v2.11.0-candidate-5 (97ee1a114aa1e5df05d86a9dd47f7fe7a2194211) > > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-5 > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc5/images/sha256-548a518bb18890ddf8e48f0bad62695cfc4f743b9e1223ca45b2e03dba6e9149 > > > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc5/images/sha256-3b755fb67d49abeb7ab6a76b7123cc474375e3881526db26f43c8cfccdaa3cf6 > > > > Please download the source package, and follow the > > release-candidate-validation doc to build > > and run the Pulsar standalone service. > > https://pulsar.apache.org/contribute/validate-release-candidate > > > > Since the metadata store is changed from ZK to RocksDB, the verification > of > > the `stateful functions` needs to set the parameter "export > > PULSAR_STANDALONE_USE_ZOOKEEPER=1" > > > > > > > > Regards > > Jiwei Guo (Tboy) >
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
I found there are both apache-pulsar-shell-2.11.0-bin.tar.gz and apache-pulsar-shell-2.11.0-bin.zip, are they duplicates? Thanks, Yunze On Mon, Jan 2, 2023 at 8:01 PM guo jiwei wrote: > > This is the fifth release candidate for Apache Pulsar, version 2.11.0. > > This release contains 1617 commits by 61 contributors. > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-5 > > CI for this release candidate > https://github.com/Technoboy-/pulsar/pull/26 > > *** Please download, test and vote on this release. This vote will stay open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-5 > > SHA-512 checksums: > > 5bf77f600ac23c7ec72696655a49801e0755d67fb4c59c7ab30e9449b061aacef61ca227f106e6b7f5f490fc2ac57412ac2af3ff064ba1e6a31c162d4ca2ba6d > > ./apache-pulsar-2.11.0-bin.tar.gz > > 0a3cab04b4a95ced3745d8b41fc3fa2632c91f77cb10e5c7e008b09ba2f279fc14de109626a44a9482b5354c5fa17bbc07ee369a32a252e358298e494c0fc1cb > > ./apache-pulsar-2.11.0-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1203/ > > The tag to be voted upon: > v2.11.0-candidate-5 (97ee1a114aa1e5df05d86a9dd47f7fe7a2194211) > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-5 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc5/images/sha256-548a518bb18890ddf8e48f0bad62695cfc4f743b9e1223ca45b2e03dba6e9149 > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc5/images/sha256-3b755fb67d49abeb7ab6a76b7123cc474375e3881526db26f43c8cfccdaa3cf6 > > Please download the source package, and follow the > release-candidate-validation doc to build > and run the Pulsar standalone service. > https://pulsar.apache.org/contribute/validate-release-candidate > > Since the metadata store is changed from ZK to RocksDB, the verification of > the `stateful functions` needs to set the parameter "export > PULSAR_STANDALONE_USE_ZOOKEEPER=1" > > > > Regards > Jiwei Guo (Tboy)
Re: [VOTE] Pulsar Release 2.11.0 Candidate-5
+1 (binding) - Checked the signature - Build from source - Checked license(server and shell) - Start standalone with zookeeper - Checked function - Checked Cassandra connector - Checked stateful function - Checked the Pulsar SQL Thanks, Penghui On Mon, Jan 2, 2023 at 8:01 PM guo jiwei wrote: > This is the fifth release candidate for Apache Pulsar, version 2.11.0. > > This release contains 1617 commits by 61 contributors. > https://github.com/apache/pulsar/compare/v2.10.2...v2.11.0-candidate-5 > > CI for this release candidate > https://github.com/Technoboy-/pulsar/pull/26 > > *** Please download, test and vote on this release. This vote will stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience. > > Source and binary files: > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.0-candidate-5 > > SHA-512 checksums: > > > 5bf77f600ac23c7ec72696655a49801e0755d67fb4c59c7ab30e9449b061aacef61ca227f106e6b7f5f490fc2ac57412ac2af3ff064ba1e6a31c162d4ca2ba6d > > ./apache-pulsar-2.11.0-bin.tar.gz > > > 0a3cab04b4a95ced3745d8b41fc3fa2632c91f77cb10e5c7e008b09ba2f279fc14de109626a44a9482b5354c5fa17bbc07ee369a32a252e358298e494c0fc1cb > > ./apache-pulsar-2.11.0-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1203/ > > The tag to be voted upon: > v2.11.0-candidate-5 (97ee1a114aa1e5df05d86a9dd47f7fe7a2194211) > https://github.com/apache/pulsar/releases/tag/v2.11.0-candidate-5 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > https://hub.docker.com/layers/mattison/pulsar-all/2.11.0-rc5/images/sha256-548a518bb18890ddf8e48f0bad62695cfc4f743b9e1223ca45b2e03dba6e9149 > > https://hub.docker.com/layers/mattison/pulsar/2.11.0-rc5/images/sha256-3b755fb67d49abeb7ab6a76b7123cc474375e3881526db26f43c8cfccdaa3cf6 > > Please download the source package, and follow the > release-candidate-validation doc to build > and run the Pulsar standalone service. > https://pulsar.apache.org/contribute/validate-release-candidate > > Since the metadata store is changed from ZK to RocksDB, the verification of > the `stateful functions` needs to set the parameter "export > PULSAR_STANDALONE_USE_ZOOKEEPER=1" > > > > Regards > Jiwei Guo (Tboy) >
Re: [ATTENTION] The source of Pulsar document is moved to apache/pulsar-site
> I'd prefer not to put too many things in the template. I agree with this sentiment. I think the template is already too busy. Many active contributors ignore important sections leaving them with the default text. However, I think a link pointing to where to contribute documentation would help new contributors. We could also point contributors to the contribution guide. I recently contributed to kedacore/keda and they had a concise template that was intuitive and had several links [0]. Thanks, Michael [0] https://raw.githubusercontent.com/kedacore/keda/main/.github/PULL_REQUEST_TEMPLATE.md On Tue, Jan 3, 2023 at 11:23 PM tison wrote: > > Thanks for your feedback! > > Michael: > > I've updated the "Edit this page" link to jump to the site repo and the > Documentation guide should properly describe the process. Do you find some > incorrect content in the PR template? Otherwise, I'd prefer not to put too > many things in the template. > > Best, > tison. > > > Michael Marshall 于2023年1月4日周三 13:09写道： > > > Thanks for all of your work on this! > > > > One follow up is that we should update the PR template [0] so that > > contributors know where to contribute documentation. > > > > Thanks, > > Michael > > > > [0] > > https://github.com/apache/pulsar/blob/9ec1d071c7188a2db694e9d7b359faaf33cb076e/.github/PULL_REQUEST_TEMPLATE.md > > > > On Tue, Jan 3, 2023 at 8:29 PM Yu wrote: > > > > > > Hi tison, thanks for your great work! > > > > > > On Fri, Dec 30, 2022 at 6:41 PM tison wrote: > > > > > > > Hi, > > > > > > > > With the resolution of [1], I moved the source of the Pulsar document > > to > > > > the site repo[2][3][4][5]. > > > > > > > > Now, if you'd like to update Pulsar document, you should submit a PR > > > > against the site repo (apache/pulsar-site) instead of the main repo > > > > (apache/pulsar, the site2 folder). > > > > > > > > I've updated the Edit URL on the page[6], so if you click the "Edit > > this > > > > page" button, you will be guided to the correct place. > > > > > > > > The docs layout is almost the same as former site2 folder. I'll update > > the > > > > corresponding contribution guide and release guide later. You can keep > > an > > > > eye on the tracking issue[7]. > > > > > > > > Best, > > > > tison. > > > > > > > > [1] https://lists.apache.org/thread/89wl112cflvtoyfg3l7kglr1dozsljm9 > > > > [2] https://github.com/apache/pulsar-site > > > > [3] https://github.com/apache/pulsar/pull/19100 > > > > [4] https://github.com/apache/pulsar-site/pull/348 > > > > [5] https://github.com/apache/pulsar/issues/19090 > > > > [6] > > > > > > > > > > https://github.com/apache/pulsar-site/commit/a2c2d23c21fd9cf936a4381901001b5889973d8e > > > > [7] https://github.com/apache/pulsar/issues/19064 > > > > > >
Re: [ATTENTION] The source of Pulsar document is moved to apache/pulsar-site
Thanks for your feedback! Michael: I've updated the "Edit this page" link to jump to the site repo and the Documentation guide should properly describe the process. Do you find some incorrect content in the PR template? Otherwise, I'd prefer not to put too many things in the template. Best, tison. Michael Marshall 于2023年1月4日周三 13:09写道： > Thanks for all of your work on this! > > One follow up is that we should update the PR template [0] so that > contributors know where to contribute documentation. > > Thanks, > Michael > > [0] > https://github.com/apache/pulsar/blob/9ec1d071c7188a2db694e9d7b359faaf33cb076e/.github/PULL_REQUEST_TEMPLATE.md > > On Tue, Jan 3, 2023 at 8:29 PM Yu wrote: > > > > Hi tison, thanks for your great work! > > > > On Fri, Dec 30, 2022 at 6:41 PM tison wrote: > > > > > Hi, > > > > > > With the resolution of [1], I moved the source of the Pulsar document > to > > > the site repo[2][3][4][5]. > > > > > > Now, if you'd like to update Pulsar document, you should submit a PR > > > against the site repo (apache/pulsar-site) instead of the main repo > > > (apache/pulsar, the site2 folder). > > > > > > I've updated the Edit URL on the page[6], so if you click the "Edit > this > > > page" button, you will be guided to the correct place. > > > > > > The docs layout is almost the same as former site2 folder. I'll update > the > > > corresponding contribution guide and release guide later. You can keep > an > > > eye on the tracking issue[7]. > > > > > > Best, > > > tison. > > > > > > [1] https://lists.apache.org/thread/89wl112cflvtoyfg3l7kglr1dozsljm9 > > > [2] https://github.com/apache/pulsar-site > > > [3] https://github.com/apache/pulsar/pull/19100 > > > [4] https://github.com/apache/pulsar-site/pull/348 > > > [5] https://github.com/apache/pulsar/issues/19090 > > > [6] > > > > > > > https://github.com/apache/pulsar-site/commit/a2c2d23c21fd9cf936a4381901001b5889973d8e > > > [7] https://github.com/apache/pulsar/issues/19064 > > > >
Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar
Congratulations, Yunze! - Michael On Tue, Jan 3, 2023 at 8:26 PM houxiaoyu wrote: > > Congrats! > > Best, > Hou xiaoyu > > Qiang Huang 于2023年1月3日周二 14:53写道： > > > Congratulations!!! Yunze > > > > > > Jun Ma 于2022年12月31日周六 17:26写道： > > > > > Congrats and well deserved! > > > > > > From: Dave Fisher > > > Sent: Saturday, December 31, 2022 1:52 > > > To: dev@pulsar.apache.org > > > Subject: Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar > > > > > > Congratulations Yunze! Well deserved! > > > > > > All the best, > > > Dave > > > > > > > > > > On Dec 29, 2022, at 4:42 AM, Haiting Jiang > > > wrote: > > > > > > > > Hi all, > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Yunze > > Xu > > > > (https://github.com/BewareMyPower) as a member of the PMC and we are > > > > pleased to announce that he has accepted. > > > > > > > > He is very active in the community in the past few years and made a lot > > > of great contributions. > > > > > > > > Welcome Yunze to the Apache Pulsar PMC. > > > > > > > > Best Regards, > > > > Haiting Jiang on behalf of the Pulsar PMC > > > > > > > > > > -- > > BR, > > Qiang Huang > >
Re: [ATTENTION] The source of Pulsar document is moved to apache/pulsar-site
Thanks for all of your work on this! One follow up is that we should update the PR template [0] so that contributors know where to contribute documentation. Thanks, Michael [0] https://github.com/apache/pulsar/blob/9ec1d071c7188a2db694e9d7b359faaf33cb076e/.github/PULL_REQUEST_TEMPLATE.md On Tue, Jan 3, 2023 at 8:29 PM Yu wrote: > > Hi tison, thanks for your great work! > > On Fri, Dec 30, 2022 at 6:41 PM tison wrote: > > > Hi, > > > > With the resolution of [1], I moved the source of the Pulsar document to > > the site repo[2][3][4][5]. > > > > Now, if you'd like to update Pulsar document, you should submit a PR > > against the site repo (apache/pulsar-site) instead of the main repo > > (apache/pulsar, the site2 folder). > > > > I've updated the Edit URL on the page[6], so if you click the "Edit this > > page" button, you will be guided to the correct place. > > > > The docs layout is almost the same as former site2 folder. I'll update the > > corresponding contribution guide and release guide later. You can keep an > > eye on the tracking issue[7]. > > > > Best, > > tison. > > > > [1] https://lists.apache.org/thread/89wl112cflvtoyfg3l7kglr1dozsljm9 > > [2] https://github.com/apache/pulsar-site > > [3] https://github.com/apache/pulsar/pull/19100 > > [4] https://github.com/apache/pulsar-site/pull/348 > > [5] https://github.com/apache/pulsar/issues/19090 > > [6] > > > > https://github.com/apache/pulsar-site/commit/a2c2d23c21fd9cf936a4381901001b5889973d8e > > [7] https://github.com/apache/pulsar/issues/19064 > >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
Modifying the subscribe request is better than exposing AUTO_CONSUME
schema type IMO. The negative value of a schema type, like BYTES,
AUTO_PRODUCE, means this schema type should only be used internally.
Adding the negative enum value to the Schema definition in
PulsarApi.proto looks very weird.
But I'm still wondering if we can avoid the API changes. I will look
deeper into this issue.
Thanks,
Yunze
On Wed, Jan 4, 2023 at 12:12 AM Enrico Olivelli wrote:
>
> Il Mar 3 Gen 2023, 14:37 Yunze Xu ha scritto:
>
> > Hi Bo,
> >
> > I got it now. The PIP title sounds ambiguous. Using the term "Upload
> > xxx SchemaType" sounds like uploading the schema into the registry.
> > Instead, it should be "carrying schema in the request when subscribing
> > with AUTO_CONSUME schema".
> >
>
>
> I agree that we should change the naming and we should probably not use a
> new Schema type but add an optional field in the subscribe request (and do
> not send it if the broker is an old version)
>
>
> Enrico
>
>
>
> > Thanks,
> > Yunze
> >
> > On Tue, Jan 3, 2023 at 4:56 PM 丛搏 wrote:
> > >
> > > Hi, Yunze
> > > > What I am concerned about is that if the old clients with other
> > > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > > the topic with AUTO_CONSUME schema, what will happen?
> > >
> > > AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> > > only represents one consumer with AUTO_CONSUME schema to subscribe to
> > > a topic. If old clients with other schemas subscribe to this topic,
> > > Its behavior will not be changed by this PIP.
> > >
> > > > What's the schema compatibility check rule on a topic with
> > AUTO_CONSUME schema?
> > >
> > > it's only the consumer schema compatibility check, not on topic. if a
> > > consume with AUTO_CONSUME schema will do any compatibility check
> > >
> > > Thanks,
> > > Bo
> > >
> > > Yunze Xu 于2023年1月3日周二 10:16写道：
> > > >
> > > > What I am concerned about is that if the old clients with other
> > > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > > the topic with AUTO_CONSUME schema, what will happen? What's the
> > > > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu
> > wrote:
> > > > >
> > > > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value
> > should
> > > > > be the same.
> > > > > 2.These changes do not affect produce and are only affect consumer
> > > > > subscribe behavior.
> > > > > 3.backward compatibility:
> > > > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME) {
> > > > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > > > .thenCompose(v -> topic.subscribe(option));
> > > > > } else {
> > > > > return topic.subscribe(option);
> > > > > }
> > > > > For the older pulsar client, the schema is null if AUTO_CONSUME
> > consumer
> > > > > subscribe to the Topic.
> > > > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the
> > Topic,
> > > > > then schema is not null and schema.getType() =
> > SchemaType.AUTO_CONSUME.
> > > > > Both new and old pulsar clients consume the topic, will return topic.
> > > > > subscribe(option).
> > > > >
> > > > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > > > #addSchemaIfIdleOrCheckCompatible.
> > > > > @Override
> > > > > public CompletableFuture
> > addSchemaIfIdleOrCheckCompatible(SchemaData
> > > > > schema) {
> > > > > return hasSchema().thenCompose((hasSchema) -> {
> > > > > int numActiveConsumersWithoutAutoSchema =
> > subscriptions.values().stream()
> > > > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > > > .filter(consumer -> consumer.getSchemaType() !=
> > SchemaType.AUTO_CONSUME)
> > > > > .toList().size())
> > > > > .sum();
> > > > > if (hasSchema
> > > > > || (!producers.isEmpty())
> > > > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > > > || (ledger.getTotalSize() != 0)) {
> > > > > return checkSchemaCompatibleForConsumer(schema);
> > > > > } else {
> > > > > return addSchema(schema).thenCompose(schemaVersion ->
> > > > > CompletableFuture.completedFuture(null));
> > > > > }
> > > > > });
> > > > > }
> > > > > Only in one case will there be a bug.
> > > > > First, the old pulsar client consume the empty topic, the consumer
> > schema
> > > > > is AUTO_CONSUME, and then whether the new or old pulsar client
> > consume(i.e.
> > > > > schema is AVRO) the topic.
> > > > > The broker will return the error message as
> > IncompatibleSchemaException ("
> > > > > Topic does not have a schema to check "). The bug at issue17354 is
> > not
> > > > > fixed in this case.
> > > > > All the other cases will be normal.
> > > > >
> > > > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > > > >
> > > > > > Defining `AutoConsume` as -3 is somehow strange. Could you
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
You mean add an optional field in the subscribe request, like (optional
bool is_autoconsume =6 [default = false];)?
Enrico Olivelli 于 2023年1月4日周三 上午12:12写道：
> Il Mar 3 Gen 2023, 14:37 Yunze Xu ha
> scritto:
>
> > Hi Bo,
> >
> > I got it now. The PIP title sounds ambiguous. Using the term "Upload
> > xxx SchemaType" sounds like uploading the schema into the registry.
> > Instead, it should be "carrying schema in the request when subscribing
> > with AUTO_CONSUME schema".
> >
>
>
> I agree that we should change the naming and we should probably not use a
> new Schema type but add an optional field in the subscribe request (and do
> not send it if the broker is an old version)
>
>
> Enrico
>
>
>
> > Thanks,
> > Yunze
> >
> > On Tue, Jan 3, 2023 at 4:56 PM 丛搏 wrote:
> > >
> > > Hi, Yunze
> > > > What I am concerned about is that if the old clients with other
> > > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > > the topic with AUTO_CONSUME schema, what will happen?
> > >
> > > AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> > > only represents one consumer with AUTO_CONSUME schema to subscribe to
> > > a topic. If old clients with other schemas subscribe to this topic,
> > > Its behavior will not be changed by this PIP.
> > >
> > > > What's the schema compatibility check rule on a topic with
> > AUTO_CONSUME schema?
> > >
> > > it's only the consumer schema compatibility check, not on topic. if a
> > > consume with AUTO_CONSUME schema will do any compatibility check
> > >
> > > Thanks,
> > > Bo
> > >
> > > Yunze Xu 于2023年1月3日周二 10:16写道：
> > > >
> > > > What I am concerned about is that if the old clients with other
> > > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > > the topic with AUTO_CONSUME schema, what will happen? What's the
> > > > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu
> > wrote:
> > > > >
> > > > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value
> > should
> > > > > be the same.
> > > > > 2.These changes do not affect produce and are only affect consumer
> > > > > subscribe behavior.
> > > > > 3.backward compatibility:
> > > > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME)
> {
> > > > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > > > .thenCompose(v -> topic.subscribe(option));
> > > > > } else {
> > > > > return topic.subscribe(option);
> > > > > }
> > > > > For the older pulsar client, the schema is null if AUTO_CONSUME
> > consumer
> > > > > subscribe to the Topic.
> > > > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the
> > Topic,
> > > > > then schema is not null and schema.getType() =
> > SchemaType.AUTO_CONSUME.
> > > > > Both new and old pulsar clients consume the topic, will return
> topic.
> > > > > subscribe(option).
> > > > >
> > > > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > > > #addSchemaIfIdleOrCheckCompatible.
> > > > > @Override
> > > > > public CompletableFuture
> > addSchemaIfIdleOrCheckCompatible(SchemaData
> > > > > schema) {
> > > > > return hasSchema().thenCompose((hasSchema) -> {
> > > > > int numActiveConsumersWithoutAutoSchema =
> > subscriptions.values().stream()
> > > > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > > > .filter(consumer -> consumer.getSchemaType() !=
> > SchemaType.AUTO_CONSUME)
> > > > > .toList().size())
> > > > > .sum();
> > > > > if (hasSchema
> > > > > || (!producers.isEmpty())
> > > > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > > > || (ledger.getTotalSize() != 0)) {
> > > > > return checkSchemaCompatibleForConsumer(schema);
> > > > > } else {
> > > > > return addSchema(schema).thenCompose(schemaVersion ->
> > > > > CompletableFuture.completedFuture(null));
> > > > > }
> > > > > });
> > > > > }
> > > > > Only in one case will there be a bug.
> > > > > First, the old pulsar client consume the empty topic, the consumer
> > schema
> > > > > is AUTO_CONSUME, and then whether the new or old pulsar client
> > consume(i.e.
> > > > > schema is AVRO) the topic.
> > > > > The broker will return the error message as
> > IncompatibleSchemaException ("
> > > > > Topic does not have a schema to check "). The bug at issue17354 is
> > not
> > > > > fixed in this case.
> > > > > All the other cases will be normal.
> > > > >
> > > > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > > > >
> > > > > > Defining `AutoConsume` as -3 is somehow strange. Could you
> clarify
> > if
> > > > > > backward compatibility is guaranteed? i.e. if the new Pulsar
> client
> > > > > > uploaded the AUTO_CONSUME schema to the broker, can the old
> Pulsar
> > > > > > clients produce or consume the same topic anymore?
> > > > > >
> > > > > > Thanks,
> > > > > > Yunze
> > > > >
Re: [ATTENTION] The source of Pulsar document is moved to apache/pulsar-site
Hi tison, thanks for your great work! On Fri, Dec 30, 2022 at 6:41 PM tison wrote: > Hi, > > With the resolution of [1], I moved the source of the Pulsar document to > the site repo[2][3][4][5]. > > Now, if you'd like to update Pulsar document, you should submit a PR > against the site repo (apache/pulsar-site) instead of the main repo > (apache/pulsar, the site2 folder). > > I've updated the Edit URL on the page[6], so if you click the "Edit this > page" button, you will be guided to the correct place. > > The docs layout is almost the same as former site2 folder. I'll update the > corresponding contribution guide and release guide later. You can keep an > eye on the tracking issue[7]. > > Best, > tison. > > [1] https://lists.apache.org/thread/89wl112cflvtoyfg3l7kglr1dozsljm9 > [2] https://github.com/apache/pulsar-site > [3] https://github.com/apache/pulsar/pull/19100 > [4] https://github.com/apache/pulsar-site/pull/348 > [5] https://github.com/apache/pulsar/issues/19090 > [6] > > https://github.com/apache/pulsar-site/commit/a2c2d23c21fd9cf936a4381901001b5889973d8e > [7] https://github.com/apache/pulsar/issues/19064 >
Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar
Congrats! Best, Hou xiaoyu Qiang Huang 于2023年1月3日周二 14:53写道： > Congratulations!!! Yunze > > > Jun Ma 于2022年12月31日周六 17:26写道： > > > Congrats and well deserved! > > > > From: Dave Fisher > > Sent: Saturday, December 31, 2022 1:52 > > To: dev@pulsar.apache.org > > Subject: Re: [ANNOUNCE] Yunze Xu as a new PMC member in Apache Pulsar > > > > Congratulations Yunze! Well deserved! > > > > All the best, > > Dave > > > > > > > On Dec 29, 2022, at 4:42 AM, Haiting Jiang > > wrote: > > > > > > Hi all, > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Yunze > Xu > > > (https://github.com/BewareMyPower) as a member of the PMC and we are > > > pleased to announce that he has accepted. > > > > > > He is very active in the community in the past few years and made a lot > > of great contributions. > > > > > > Welcome Yunze to the Apache Pulsar PMC. > > > > > > Best Regards, > > > Haiting Jiang on behalf of the Pulsar PMC > > > > > > -- > BR, > Qiang Huang >
Re: [Discuss] Deprecate Index-based Publisher Stat Aggregation in Topics Partitioned-Stats
Hi Pulsar dev, I raised a PIP for the default config change(change the aggregatePublisherStatsByProducerName config default to true) as requested. PIP: https://github.com/apache/pulsar/issues/19125 I guess we can continue to use this discussion thread for this PIP as well. Thank you, Heesung On Fri, Dec 9, 2022 at 2:12 PM Heesung Sohn wrote: > Hi, > > I raised a PR to the master branch here, > https://github.com/apache/pulsar/pull/18807. > > PTAL. > > Thank you, > Heesung > > On Wed, Nov 30, 2022 at 4:06 AM Enrico Olivelli > wrote: > >> Heesung, >> I also agree that we must preserve compatibility, if we want to pick >> this change to released versions. >> For a new major version we COULD break compatibility only as a last >> resort. >> >> If we have a way to improve the code and use less resources, I believe >> that it is the best way. >> >> Enrico >> >> Il giorno mer 30 nov 2022 alle ore 11:56 Heesung Sohn >> ha scritto: >> > >> > Hi, >> > >> > I raised this PR to my local fork to sync our direction. >> > >> > https://github.com/heesung-sn/pulsar/pull/15/files >> > >> > Please let me know if this direction looks good. Then, I will proceed to >> > raise an official PR. >> > >> > >> > Thanks, >> > Heesung >> >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
Il Mar 3 Gen 2023, 14:37 Yunze Xu ha scritto:
> Hi Bo,
>
> I got it now. The PIP title sounds ambiguous. Using the term "Upload
> xxx SchemaType" sounds like uploading the schema into the registry.
> Instead, it should be "carrying schema in the request when subscribing
> with AUTO_CONSUME schema".
>
I agree that we should change the naming and we should probably not use a
new Schema type but add an optional field in the subscribe request (and do
not send it if the broker is an old version)
Enrico
> Thanks,
> Yunze
>
> On Tue, Jan 3, 2023 at 4:56 PM 丛搏 wrote:
> >
> > Hi, Yunze
> > > What I am concerned about is that if the old clients with other
> > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > the topic with AUTO_CONSUME schema, what will happen?
> >
> > AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> > only represents one consumer with AUTO_CONSUME schema to subscribe to
> > a topic. If old clients with other schemas subscribe to this topic,
> > Its behavior will not be changed by this PIP.
> >
> > > What's the schema compatibility check rule on a topic with
> AUTO_CONSUME schema?
> >
> > it's only the consumer schema compatibility check, not on topic. if a
> > consume with AUTO_CONSUME schema will do any compatibility check
> >
> > Thanks,
> > Bo
> >
> > Yunze Xu 于2023年1月3日周二 10:16写道：
> > >
> > > What I am concerned about is that if the old clients with other
> > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > the topic with AUTO_CONSUME schema, what will happen? What's the
> > > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu
> wrote:
> > > >
> > > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value
> should
> > > > be the same.
> > > > 2.These changes do not affect produce and are only affect consumer
> > > > subscribe behavior.
> > > > 3.backward compatibility:
> > > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME) {
> > > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > > .thenCompose(v -> topic.subscribe(option));
> > > > } else {
> > > > return topic.subscribe(option);
> > > > }
> > > > For the older pulsar client, the schema is null if AUTO_CONSUME
> consumer
> > > > subscribe to the Topic.
> > > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the
> Topic,
> > > > then schema is not null and schema.getType() =
> SchemaType.AUTO_CONSUME.
> > > > Both new and old pulsar clients consume the topic, will return topic.
> > > > subscribe(option).
> > > >
> > > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > > #addSchemaIfIdleOrCheckCompatible.
> > > > @Override
> > > > public CompletableFuture
> addSchemaIfIdleOrCheckCompatible(SchemaData
> > > > schema) {
> > > > return hasSchema().thenCompose((hasSchema) -> {
> > > > int numActiveConsumersWithoutAutoSchema =
> subscriptions.values().stream()
> > > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > > .filter(consumer -> consumer.getSchemaType() !=
> SchemaType.AUTO_CONSUME)
> > > > .toList().size())
> > > > .sum();
> > > > if (hasSchema
> > > > || (!producers.isEmpty())
> > > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > > || (ledger.getTotalSize() != 0)) {
> > > > return checkSchemaCompatibleForConsumer(schema);
> > > > } else {
> > > > return addSchema(schema).thenCompose(schemaVersion ->
> > > > CompletableFuture.completedFuture(null));
> > > > }
> > > > });
> > > > }
> > > > Only in one case will there be a bug.
> > > > First, the old pulsar client consume the empty topic, the consumer
> schema
> > > > is AUTO_CONSUME, and then whether the new or old pulsar client
> consume(i.e.
> > > > schema is AVRO) the topic.
> > > > The broker will return the error message as
> IncompatibleSchemaException ("
> > > > Topic does not have a schema to check "). The bug at issue17354 is
> not
> > > > fixed in this case.
> > > > All the other cases will be normal.
> > > >
> > > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > > >
> > > > > Defining `AutoConsume` as -3 is somehow strange. Could you clarify
> if
> > > > > backward compatibility is guaranteed? i.e. if the new Pulsar client
> > > > > uploaded the AUTO_CONSUME schema to the broker, can the old Pulsar
> > > > > clients produce or consume the same topic anymore?
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Fri, Dec 30, 2022 at 11:32 PM 思楠刘
> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I made a PIP to discuss:
> https://github.com/apache/pulsar/issues/19113.
> > > > > >
> > > > > > Thanks,
> > > > > > Sinan
> > > > >
>
[DISCUSS] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext
Hi all, I opened a PIP to discuss making PulsarAdmin accessible in Pulsar IO connectors through SinkContext and SourceContext: https://github.com/apache/pulsar/issues/19123 Looking forward to hearing your thoughts, Alex
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
Hi Bo,
I got it now. The PIP title sounds ambiguous. Using the term "Upload
xxx SchemaType" sounds like uploading the schema into the registry.
Instead, it should be "carrying schema in the request when subscribing
with AUTO_CONSUME schema".
Thanks,
Yunze
On Tue, Jan 3, 2023 at 4:56 PM 丛搏 wrote:
>
> Hi, Yunze
> > What I am concerned about is that if the old clients with other
> > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > the topic with AUTO_CONSUME schema, what will happen?
>
> AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> only represents one consumer with AUTO_CONSUME schema to subscribe to
> a topic. If old clients with other schemas subscribe to this topic,
> Its behavior will not be changed by this PIP.
>
> > What's the schema compatibility check rule on a topic with AUTO_CONSUME
> > schema?
>
> it's only the consumer schema compatibility check, not on topic. if a
> consume with AUTO_CONSUME schema will do any compatibility check
>
> Thanks,
> Bo
>
> Yunze Xu 于2023年1月3日周二 10:16写道：
> >
> > What I am concerned about is that if the old clients with other
> > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > the topic with AUTO_CONSUME schema, what will happen? What's the
> > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> >
> > Thanks,
> > Yunze
> >
> > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu wrote:
> > >
> > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value should
> > > be the same.
> > > 2.These changes do not affect produce and are only affect consumer
> > > subscribe behavior.
> > > 3.backward compatibility:
> > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME) {
> > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > .thenCompose(v -> topic.subscribe(option));
> > > } else {
> > > return topic.subscribe(option);
> > > }
> > > For the older pulsar client, the schema is null if AUTO_CONSUME consumer
> > > subscribe to the Topic.
> > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the Topic,
> > > then schema is not null and schema.getType() = SchemaType.AUTO_CONSUME.
> > > Both new and old pulsar clients consume the topic, will return topic.
> > > subscribe(option).
> > >
> > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > #addSchemaIfIdleOrCheckCompatible.
> > > @Override
> > > public CompletableFuture addSchemaIfIdleOrCheckCompatible(SchemaData
> > > schema) {
> > > return hasSchema().thenCompose((hasSchema) -> {
> > > int numActiveConsumersWithoutAutoSchema = subscriptions.values().stream()
> > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > .filter(consumer -> consumer.getSchemaType() != SchemaType.AUTO_CONSUME)
> > > .toList().size())
> > > .sum();
> > > if (hasSchema
> > > || (!producers.isEmpty())
> > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > || (ledger.getTotalSize() != 0)) {
> > > return checkSchemaCompatibleForConsumer(schema);
> > > } else {
> > > return addSchema(schema).thenCompose(schemaVersion ->
> > > CompletableFuture.completedFuture(null));
> > > }
> > > });
> > > }
> > > Only in one case will there be a bug.
> > > First, the old pulsar client consume the empty topic, the consumer schema
> > > is AUTO_CONSUME, and then whether the new or old pulsar client
> > > consume(i.e.
> > > schema is AVRO) the topic.
> > > The broker will return the error message as IncompatibleSchemaException ("
> > > Topic does not have a schema to check "). The bug at issue17354 is not
> > > fixed in this case.
> > > All the other cases will be normal.
> > >
> > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > >
> > > > Defining `AutoConsume` as -3 is somehow strange. Could you clarify if
> > > > backward compatibility is guaranteed? i.e. if the new Pulsar client
> > > > uploaded the AUTO_CONSUME schema to the broker, can the old Pulsar
> > > > clients produce or consume the same topic anymore?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Fri, Dec 30, 2022 at 11:32 PM 思楠刘 wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I made a PIP to discuss:
> > > > > https://github.com/apache/pulsar/issues/19113.
> > > > >
> > > > > Thanks,
> > > > > Sinan
> > > >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
If we use a consumer with AUTO_CONSUME schema to subscribe to a topic,
there will be problems. Because the schema is not null, so the consumer
with AUTO_CONSUME schema will not directly return topic.subscribe(option).
This will result in check Schema Compatible or addSchema.
1. "errorMsg":"Incompatible schema: exists schema type AVRO, new schema
type AUTO_CONSUME"
https://github.com/apache/pulsar/blob/9ec1d071c7188a2db694e9d7b359faaf33cb076e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java#L468-L471
2. "errorMsg":"Schema AUTO_CONSUME is a client-side schema type"
https://github.com/apache/pulsar/blob/9ec1d071c7188a2db694e9d7b359faaf33cb076e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/validator/SchemaDataValidator.java#L70-L74
Enrico Olivelli 于2023年1月3日周二 18:46写道：
> (I apologise for top posting)
> What happens if a new client connects to a old broker?
>
> Enrico
>
> Il Mar 3 Gen 2023, 10:51 丛搏 ha scritto:
>
> > +1 (non-binding)
> >
> > Thanks,
> > Bo
> >
> > 丛搏 于2023年1月3日周二 16:56写道：
> > >
> > > Hi, Yunze
> > > > What I am concerned about is that if the old clients with other
> > > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > > the topic with AUTO_CONSUME schema, what will happen?
> > >
> > > AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> > > only represents one consumer with AUTO_CONSUME schema to subscribe to
> > > a topic. If old clients with other schemas subscribe to this topic,
> > > Its behavior will not be changed by this PIP.
> > >
> > > > What's the schema compatibility check rule on a topic with
> > AUTO_CONSUME schema?
> > >
> > > it's only the consumer schema compatibility check, not on topic. if a
> > > consume with AUTO_CONSUME schema will do any compatibility check
> > >
> > > Thanks,
> > > Bo
> > >
> > > Yunze Xu 于2023年1月3日周二 10:16写道：
> > > >
> > > > What I am concerned about is that if the old clients with other
> > > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > > the topic with AUTO_CONSUME schema, what will happen? What's the
> > > > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu
> > wrote:
> > > > >
> > > > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value
> > should
> > > > > be the same.
> > > > > 2.These changes do not affect produce and are only affect consumer
> > > > > subscribe behavior.
> > > > > 3.backward compatibility:
> > > > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME)
> {
> > > > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > > > .thenCompose(v -> topic.subscribe(option));
> > > > > } else {
> > > > > return topic.subscribe(option);
> > > > > }
> > > > > For the older pulsar client, the schema is null if AUTO_CONSUME
> > consumer
> > > > > subscribe to the Topic.
> > > > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the
> > Topic,
> > > > > then schema is not null and schema.getType() =
> > SchemaType.AUTO_CONSUME.
> > > > > Both new and old pulsar clients consume the topic, will return
> topic.
> > > > > subscribe(option).
> > > > >
> > > > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > > > #addSchemaIfIdleOrCheckCompatible.
> > > > > @Override
> > > > > public CompletableFuture
> > addSchemaIfIdleOrCheckCompatible(SchemaData
> > > > > schema) {
> > > > > return hasSchema().thenCompose((hasSchema) -> {
> > > > > int numActiveConsumersWithoutAutoSchema =
> > subscriptions.values().stream()
> > > > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > > > .filter(consumer -> consumer.getSchemaType() !=
> > SchemaType.AUTO_CONSUME)
> > > > > .toList().size())
> > > > > .sum();
> > > > > if (hasSchema
> > > > > || (!producers.isEmpty())
> > > > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > > > || (ledger.getTotalSize() != 0)) {
> > > > > return checkSchemaCompatibleForConsumer(schema);
> > > > > } else {
> > > > > return addSchema(schema).thenCompose(schemaVersion ->
> > > > > CompletableFuture.completedFuture(null));
> > > > > }
> > > > > });
> > > > > }
> > > > > Only in one case will there be a bug.
> > > > > First, the old pulsar client consume the empty topic, the consumer
> > schema
> > > > > is AUTO_CONSUME, and then whether the new or old pulsar client
> > consume(i.e.
> > > > > schema is AVRO) the topic.
> > > > > The broker will return the error message as
> > IncompatibleSchemaException ("
> > > > > Topic does not have a schema to check "). The bug at issue17354 is
> > not
> > > > > fixed in this case.
> > > > > All the other cases will be normal.
> > > > >
> > > > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > > > >
> > > > > > Defining `AutoConsume` as -3 is somehow strange. Could you
>
[ANNOUNCE] Apache Pulsar 2.9.4 released
The Apache Pulsar team is proud to announce Apache Pulsar version 2.9.4. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar release details and downloads, visit: https://pulsar.apache.org/download Release Notes are at:https://pulsar.apache.org/release-notes/ We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
(I apologise for top posting)
What happens if a new client connects to a old broker?
Enrico
Il Mar 3 Gen 2023, 10:51 丛搏 ha scritto:
> +1 (non-binding)
>
> Thanks,
> Bo
>
> 丛搏 于2023年1月3日周二 16:56写道：
> >
> > Hi, Yunze
> > > What I am concerned about is that if the old clients with other
> > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > the topic with AUTO_CONSUME schema, what will happen?
> >
> > AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> > only represents one consumer with AUTO_CONSUME schema to subscribe to
> > a topic. If old clients with other schemas subscribe to this topic,
> > Its behavior will not be changed by this PIP.
> >
> > > What's the schema compatibility check rule on a topic with
> AUTO_CONSUME schema?
> >
> > it's only the consumer schema compatibility check, not on topic. if a
> > consume with AUTO_CONSUME schema will do any compatibility check
> >
> > Thanks,
> > Bo
> >
> > Yunze Xu 于2023年1月3日周二 10:16写道：
> > >
> > > What I am concerned about is that if the old clients with other
> > > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > > the topic with AUTO_CONSUME schema, what will happen? What's the
> > > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu
> wrote:
> > > >
> > > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value
> should
> > > > be the same.
> > > > 2.These changes do not affect produce and are only affect consumer
> > > > subscribe behavior.
> > > > 3.backward compatibility:
> > > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME) {
> > > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > > .thenCompose(v -> topic.subscribe(option));
> > > > } else {
> > > > return topic.subscribe(option);
> > > > }
> > > > For the older pulsar client, the schema is null if AUTO_CONSUME
> consumer
> > > > subscribe to the Topic.
> > > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the
> Topic,
> > > > then schema is not null and schema.getType() =
> SchemaType.AUTO_CONSUME.
> > > > Both new and old pulsar clients consume the topic, will return topic.
> > > > subscribe(option).
> > > >
> > > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > > #addSchemaIfIdleOrCheckCompatible.
> > > > @Override
> > > > public CompletableFuture
> addSchemaIfIdleOrCheckCompatible(SchemaData
> > > > schema) {
> > > > return hasSchema().thenCompose((hasSchema) -> {
> > > > int numActiveConsumersWithoutAutoSchema =
> subscriptions.values().stream()
> > > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > > .filter(consumer -> consumer.getSchemaType() !=
> SchemaType.AUTO_CONSUME)
> > > > .toList().size())
> > > > .sum();
> > > > if (hasSchema
> > > > || (!producers.isEmpty())
> > > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > > || (ledger.getTotalSize() != 0)) {
> > > > return checkSchemaCompatibleForConsumer(schema);
> > > > } else {
> > > > return addSchema(schema).thenCompose(schemaVersion ->
> > > > CompletableFuture.completedFuture(null));
> > > > }
> > > > });
> > > > }
> > > > Only in one case will there be a bug.
> > > > First, the old pulsar client consume the empty topic, the consumer
> schema
> > > > is AUTO_CONSUME, and then whether the new or old pulsar client
> consume(i.e.
> > > > schema is AVRO) the topic.
> > > > The broker will return the error message as
> IncompatibleSchemaException ("
> > > > Topic does not have a schema to check "). The bug at issue17354 is
> not
> > > > fixed in this case.
> > > > All the other cases will be normal.
> > > >
> > > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > > >
> > > > > Defining `AutoConsume` as -3 is somehow strange. Could you clarify
> if
> > > > > backward compatibility is guaranteed? i.e. if the new Pulsar client
> > > > > uploaded the AUTO_CONSUME schema to the broker, can the old Pulsar
> > > > > clients produce or consume the same topic anymore?
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Fri, Dec 30, 2022 at 11:32 PM 思楠刘
> wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I made a PIP to discuss:
> https://github.com/apache/pulsar/issues/19113.
> > > > > >
> > > > > > Thanks,
> > > > > > Sinan
> > > > >
>
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
+1 (non-binding)
Thanks,
Bo
丛搏 于2023年1月3日周二 16:56写道：
>
> Hi, Yunze
> > What I am concerned about is that if the old clients with other
> > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > the topic with AUTO_CONSUME schema, what will happen?
>
> AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
> only represents one consumer with AUTO_CONSUME schema to subscribe to
> a topic. If old clients with other schemas subscribe to this topic,
> Its behavior will not be changed by this PIP.
>
> > What's the schema compatibility check rule on a topic with AUTO_CONSUME
> > schema?
>
> it's only the consumer schema compatibility check, not on topic. if a
> consume with AUTO_CONSUME schema will do any compatibility check
>
> Thanks,
> Bo
>
> Yunze Xu 于2023年1月3日周二 10:16写道：
> >
> > What I am concerned about is that if the old clients with other
> > schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> > the topic with AUTO_CONSUME schema, what will happen? What's the
> > schema compatibility check rule on a topic with AUTO_CONSUME schema?
> >
> > Thanks,
> > Yunze
> >
> > On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu wrote:
> > >
> > > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value should
> > > be the same.
> > > 2.These changes do not affect produce and are only affect consumer
> > > subscribe behavior.
> > > 3.backward compatibility:
> > > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME) {
> > > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > > .thenCompose(v -> topic.subscribe(option));
> > > } else {
> > > return topic.subscribe(option);
> > > }
> > > For the older pulsar client, the schema is null if AUTO_CONSUME consumer
> > > subscribe to the Topic.
> > > For the new pulsar client, if AUTO_CONSUME consumer subscribe the Topic,
> > > then schema is not null and schema.getType() = SchemaType.AUTO_CONSUME.
> > > Both new and old pulsar clients consume the topic, will return topic.
> > > subscribe(option).
> > >
> > > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > > #addSchemaIfIdleOrCheckCompatible.
> > > @Override
> > > public CompletableFuture addSchemaIfIdleOrCheckCompatible(SchemaData
> > > schema) {
> > > return hasSchema().thenCompose((hasSchema) -> {
> > > int numActiveConsumersWithoutAutoSchema = subscriptions.values().stream()
> > > .mapToInt(subscription -> subscription.getConsumers().stream()
> > > .filter(consumer -> consumer.getSchemaType() != SchemaType.AUTO_CONSUME)
> > > .toList().size())
> > > .sum();
> > > if (hasSchema
> > > || (!producers.isEmpty())
> > > || (numActiveConsumersWithoutAutoSchema != 0)
> > > || (ledger.getTotalSize() != 0)) {
> > > return checkSchemaCompatibleForConsumer(schema);
> > > } else {
> > > return addSchema(schema).thenCompose(schemaVersion ->
> > > CompletableFuture.completedFuture(null));
> > > }
> > > });
> > > }
> > > Only in one case will there be a bug.
> > > First, the old pulsar client consume the empty topic, the consumer schema
> > > is AUTO_CONSUME, and then whether the new or old pulsar client
> > > consume(i.e.
> > > schema is AVRO) the topic.
> > > The broker will return the error message as IncompatibleSchemaException ("
> > > Topic does not have a schema to check "). The bug at issue17354 is not
> > > fixed in this case.
> > > All the other cases will be normal.
> > >
> > > Yunze Xu 于2022年12月31日周六 20:23写道：
> > >
> > > > Defining `AutoConsume` as -3 is somehow strange. Could you clarify if
> > > > backward compatibility is guaranteed? i.e. if the new Pulsar client
> > > > uploaded the AUTO_CONSUME schema to the broker, can the old Pulsar
> > > > clients produce or consume the same topic anymore?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Fri, Dec 30, 2022 at 11:32 PM 思楠刘 wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I made a PIP to discuss:
> > > > > https://github.com/apache/pulsar/issues/19113.
> > > > >
> > > > > Thanks,
> > > > > Sinan
> > > >
Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker
Hi, Yunze
> What I am concerned about is that if the old clients with other
> schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> the topic with AUTO_CONSUME schema, what will happen?
AUTO_CONSUME schema will not store in `SchemaRegistryServiceImpl`, it
only represents one consumer with AUTO_CONSUME schema to subscribe to
a topic. If old clients with other schemas subscribe to this topic,
Its behavior will not be changed by this PIP.
> What's the schema compatibility check rule on a topic with AUTO_CONSUME
> schema?
it's only the consumer schema compatibility check, not on topic. if a
consume with AUTO_CONSUME schema will do any compatibility check
Thanks,
Bo
Yunze Xu 于2023年1月3日周二 10:16写道：
>
> What I am concerned about is that if the old clients with other
> schemas (i.e. schema is neither null nor AUTO_CONSUME) subscribe to
> the topic with AUTO_CONSUME schema, what will happen? What's the
> schema compatibility check rule on a topic with AUTO_CONSUME schema?
>
> Thanks,
> Yunze
>
> On Mon, Jan 2, 2023 at 12:38 AM SiNan Liu wrote:
> >
> > 1.Schema.Type and org.apache.pulsar.common.schema.SchemaType value should
> > be the same.
> > 2.These changes do not affect produce and are only affect consumer
> > subscribe behavior.
> > 3.backward compatibility:
> > (1)In org.apache.pulsar.broker.service.ServerCnx#handleSubscribe.
> > if (schema != null && schema.getType() != SchemaType.AUTO_CONSUME) {
> > return topic.addSchemaIfIdleOrCheckCompatible(schema)
> > .thenCompose(v -> topic.subscribe(option));
> > } else {
> > return topic.subscribe(option);
> > }
> > For the older pulsar client, the schema is null if AUTO_CONSUME consumer
> > subscribe to the Topic.
> > For the new pulsar client, if AUTO_CONSUME consumer subscribe the Topic,
> > then schema is not null and schema.getType() = SchemaType.AUTO_CONSUME.
> > Both new and old pulsar clients consume the topic, will return topic.
> > subscribe(option).
> >
> > (2)In org.apache.pulsar.broker.service.persistent.PersistentTopic
> > #addSchemaIfIdleOrCheckCompatible.
> > @Override
> > public CompletableFuture addSchemaIfIdleOrCheckCompatible(SchemaData
> > schema) {
> > return hasSchema().thenCompose((hasSchema) -> {
> > int numActiveConsumersWithoutAutoSchema = subscriptions.values().stream()
> > .mapToInt(subscription -> subscription.getConsumers().stream()
> > .filter(consumer -> consumer.getSchemaType() != SchemaType.AUTO_CONSUME)
> > .toList().size())
> > .sum();
> > if (hasSchema
> > || (!producers.isEmpty())
> > || (numActiveConsumersWithoutAutoSchema != 0)
> > || (ledger.getTotalSize() != 0)) {
> > return checkSchemaCompatibleForConsumer(schema);
> > } else {
> > return addSchema(schema).thenCompose(schemaVersion ->
> > CompletableFuture.completedFuture(null));
> > }
> > });
> > }
> > Only in one case will there be a bug.
> > First, the old pulsar client consume the empty topic, the consumer schema
> > is AUTO_CONSUME, and then whether the new or old pulsar client consume(i.e.
> > schema is AVRO) the topic.
> > The broker will return the error message as IncompatibleSchemaException ("
> > Topic does not have a schema to check "). The bug at issue17354 is not
> > fixed in this case.
> > All the other cases will be normal.
> >
> > Yunze Xu 于2022年12月31日周六 20:23写道：
> >
> > > Defining `AutoConsume` as -3 is somehow strange. Could you clarify if
> > > backward compatibility is guaranteed? i.e. if the new Pulsar client
> > > uploaded the AUTO_CONSUME schema to the broker, can the old Pulsar
> > > clients produce or consume the same topic anymore?
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Fri, Dec 30, 2022 at 11:32 PM 思楠刘 wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I made a PIP to discuss: https://github.com/apache/pulsar/issues/19113.
> > > >
> > > > Thanks,
> > > > Sinan
> > >
Re: [VOTE] Pulsar Node.js Client Release 1.8.0 Candidate 2
Thank you for verifying this release. Closing the vote 3 binding +1s: * PengHui Li * Yunze Xu * Hang Chen 2 non-binding +1s: * Baodi Shi * Ran Gao > However, I'm still wondering whether we still need to upload the `napi-xxx.tar.gz` files? These files seem totally useless for users, no matter in verification or using. We should not upload these files to our release SVN repo. When we install the pulsar nodejs client using npm, the npm needs to download these files from https://archive.apache.org/. Without these files, we cannot run `npm install` correctly. We upload these files to svn only because they are necessary for the npm package installation, so they are part of the release. However, doing so also creates some inconvenience for the release process. I was wondering if we could put these files(napi-xxx.tar.gz) in the npm package file. Do you have any ideas? Thanks, Zike Yang On Fri, Dec 30, 2022 at 3:57 PM Baodi Shi wrote: > > Supplement check: > > - Node.js version 10.24.1 > - Ubuntu 20.04 > - Install the NPM package via `npm install pulsar-client@1.8.0-rc.2` > - Verify the producer and consumer. > > > > Thanks, > Baodi Shi > > > 2022年12月30日 15:11，Hang Chen 写道： > > > > +1 (binding) > > > > - Check signatures > > - Install from source on ubuntu 20.04 (node version >= 16.18 and npm > > version >= 8.19.2) > > - Run node producer and consumer to pub-sub messages > > > > Thanks, > > Hang > > > > Yunze Xu 于2022年12月30日周五 10:54写道： > >> > >> +1 (binding) > >> - Check signatures > >> - Install from source > >> - Install the NPM package via `npm install pulsar-client@1.8.0-rc.2` > >> - Verify the producer, consumer, custom_logger examples with both two > >> installation methods bove > >> > >> > >> > >> However, I'm still wondering whether we still need to upload the > >> `napi-xxx.tar.gz` files? These files seem totally useless for users, > >> no matter in verification or using. We should not upload these files > >> to our release SVN repo. > >> > >> Thanks, > >> Yunze > >> > >> On Wed, Dec 28, 2022 at 9:14 PM PengHui Li wrote: > >>> > >>> +1 (binding) > >>> > >>> Verify following the steps from > >>> https://github.com/RobertIndie/pulsar-client-node-validation > >>> - Start standalone (master branch) > >>> - Start consumer (node ./examples/consumer.js) > >>> - Start producer (node ./examples/producer.js) > >>> - The consumer can receive the messages > >>> > >>> Regards, > >>> Penghui > >>> > >>> > >>> On Wed, Dec 28, 2022 at 1:45 PM Ran Gao wrote: > >>> > +1 (non-binding) > > - Checked npm package > https://www.npmjs.com/package/pulsar-client/v/1.8.0-rc.2 > - Checked source code + napi-darwin-unknown-arm64 work on macOS 12.6 (M1 > Pro chip) > > Thanks, > Ran Gao > > On 2022/12/27 08:29:35 Zike Yang wrote: > > Hi everyone, > > > > This is the second release candidate for the Apache Pulsar Node.js > > client, version 1.8.0. > > > > It fixes the following issues: > > https://github.com/apache/pulsar-client-node/milestone/13?closed=1 > > > > Please download the source files and review this release candidate: > > - Download the source package, verify shasum and asc > > - Follow the README.md to build and run the Pulsar Node.js client. > > > > The rc package has been published to the npm registry: > > https://www.npmjs.com/package/pulsar-client/v/1.8.0-rc.2 > > You can install it by `npm i pulsar-client@1.8.0-rc.2` and verify the > package. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Source files: > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.0-candidate-2/ > > > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > > > SHA-512 checksum: > > > 80f9de8ca73b9a0bc7d083c6ba92e86721d381aa0689858270e074ff505f887533c40c5625b85a8b0943ced352e93eaafe67eb9bcb5afefadb4bd29b594620c4 > > pulsar-client-node-1.8.0.tar.gz > > > > The tag to be voted upon: > > v1.8.0-rc.2 > > https://github.com/apache/pulsar-client-node/releases/tag/v1.8.0-rc.2 > > > > Please review and vote on the release candidate #2 for the version > > 1.8.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > Thanks, > > Zike Yang > > > >
