Re: [DISCUSS] Cherry-pick #18518 into branch-2.10 and branch-2.11

[mattisonchao]

+1

Best,
Mattison
On 5 Sep 2023 at 10:00 +0800, PengHui Li , wrote:
> +1
>
> Thanks,
> Penghui
>
> On Fri, Sep 1, 2023 at 3:04 AM Yubiao Feng
>  wrote:
>
> > Cherry-picked.
> >
> > Since the method `isPartitionedTopicBeingDeletedAsync` only exists in
> > `branch-3.0`, the change
> >
> > https://github.com/apache/pulsar/pull/18518/files#diff-38fac527320b93f0e9ea2459420f56638299627f7f29c01484f7e4aae0c5f0b7R330
> > was not be cherry-picked into `branch-2.10` and `branch-2.11`
> >
> > Thanks
> > Yubiao Feng
> >
> > On Wed, Aug 30, 2023 at 1:37 AM Yubiao Feng 
> > wrote:
> >
> > > > Hi all
> > > >
> > > > ### Background
> > > > The PR https://github.com/apache/pulsar/pull/18518 added to
> > MetadataStore
> > > > the support for using the sync() API and added refreshAndGetAsync() to
> > > > MetadataCache.
> > > >
> > > > The PR https://github.com/apache/pulsar/pull/21063 fixed a bug that
> > > > Broker will create both non-partitioned topic and partitioned topic at
> > the
> > > > same time when users call `pulsar-admin create-partitioned-topic  > > > topic>`. And this PR relies on the PR #18518.
> > > >
> > > > ### Cherry-pick #18518 into branch-2.10 and branch-2.11
> > > > Since the bug which https://github.com/apache/pulsar/pull/21063 fixed
> > > > causes message may be mistakenly sent to the non-partitioned topic, but
> > no
> > > > consumer consumes it, which means that the message is lost from the
> > user's
> > > > point of view. So this is an important fix.
> > > >
> > > > Since this PR 21063 relies on PR #18518, I want to cherry-pick #18518
> > into
> > > > branch 2.10 and branch 2.11.
> > > >
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> > > >
> >

Re: [VOTE] PIP-297: Support terminating Function & Connector with the fatal exception

[mattisonchao]

+1 (binding)

Best,
Mattison
On 5 Sep 2023 at 09:07 +0800, PengHui Li , wrote:
> +1 (binding)
>
> Thanks,
> Penghui
>
> On Tue, Sep 5, 2023 at 8:40 AM Baodi Shi  wrote:
>
> > +1(non-binding)
> >
> >
> > Thanks,
> > Baodi Shi
> >
> >
> > On Sep 5, 2023 at 05:23:38, Enrico Olivelli  wrote:
> >
> > > > +1 (binding)
> > > >
> > > > Enrico
> > > >
> > > > Il giorno lun 4 set 2023 alle ore 12:32 Zike Yang  ha
> > > > scritto:
> > > >
> > > >
> > > > Hi, all
> > > >
> > > >
> > > > Since there are no other concerns in the discussion, I'm delighted to
> > > >
> > > > start the voting process for the PIP-297.
> > > >
> > > >
> > > > Here is the link to the PIP: https://github.com/apache/pulsar/pull/21079
> > > >
> > > >
> > > > BR,
> > > >
> > > > Zike Yang
> > > >
> > > >
> >

Re: [VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader

[mattisonchao]

+1(binding)

Best,
Mattison
On 28 Aug 2023 at 08:53 +0800, PengHui Li , wrote:
> +1 (binding)
>
> Thanks,
> Penghui
>
> On Fri, Aug 25, 2023 at 8:09 PM Yubiao Feng
>  wrote:
>
> > +1 (non-binding)
> >
> > Thanks
> > Yubiao Feng
> >
> > On Fri, Aug 25, 2023 at 2:53 PM Xiangying Meng 
> > wrote:
> >
> > > > Hi Pulsar Community,
> > > >
> > > > This is the vote thread for PIP 296:
> > > > https://github.com/apache/pulsar/pull/21052
> > > >
> > > > This PIP will help to improve the flexibility of Reader usage.
> > > >
> > > > Thanks,
> > > > Xiangying
> > > >
> >

Re: [DICUSS] Topic-Partition Auto Creation after Partitioned Topic Force-Removal

[mattisonchao]

Hi, Heesung Sohn

After PIP https://github.com/apache/pulsar/issues/19239 is implemented, the 
topic name includes `-partition-` keywords without metadata that will be 
rejected.  :)

Best,
Mattison
On 24 Aug 2023 at 07:58 +0800, Heesung Sohn 
, wrote:
> Hi dev,
>
> I am proposing a behavior change in Pulsar, and I wonder if this is
> acceptable as a "bug fix."
>
> PR: https://github.com/apache/pulsar/pull/20554
>
> Given that the topic-auto-creation is enabled,
> When a partitioned topic is deleted with force=true, the
> existing clients(consumers and producers) automatically
> create(recover) topic partitions after reconnecting. This results in an odd
> metadata inconsistent state, where there is
> no partitioned topic, but its topic partitions exist -- so the partitioned
> topic creation fails.
> I think this behavior is counter-intuitive and appears to be a bug(but this
> behavior is asserted by unit tests such as
> testPartitionedTopicAutoCreation, testCompatibilityWithPartitionKeyword)
> Please check the description in the PR to see the repro steps.
>
>
> Proposals
>
> Option1 :
> I think, ideally, we shouldn't allow this automatic topic partition
> creation from the clients when the root topic(the partitioned topic) is
> already deleted.
> To create the topic partitions, the admin needs to create the partitioned
> topic again.
>
> Option2 :
> Upon recreating topic partitions, it should recreate the corresponding
> partitioned topic, if not there, to make it consistent.
>
>
> Thanks,
> Heesung

Re: [VOTE] PIP-290 Support WSS E2E encryption and not need to expose the private key to the WebSocket Proxy

[mattisonchao]

+1 (binding)

This proposal looks great to me. But I've got several concerns which will not 
affect this PIP voting.

1. You mixed compression and E2E encryption support in one proposal. I am 
unsure if we should split them into two parts(compression & E2E) to help make 
the proposal not too complex. but anyway. IMO, it's better to split the 
implementation PR into multiple that will help the reviewer review this PR more 
easily.
2. Actually, We uses a tricky way to avoid client don't compress the compressed 
data again(We set producer compression type to NONE. but we use the 
ProducerImpl to send a message entity with compressed data). IMO, it's better 
to introduce a public API to help client support it. but it's fine to use it to 
solve the web socket problem now.



Best,
Mattison
On 22 Aug 2023 at 11:16 +0800, PengHui Li , wrote:
> +1(binding)
>
> - The motivation looks good to me. The proposal will provide a real e2e
> encryption solution for the WebSocket proxy
> - The solution looks good to me. It will not introduce break changes and
> will use public APIs as much as possible. And it will not introduce any
> extra configuration. The API definition is clear and aligns with the
> existing naming pattern.
> - For the public API changes. We already have an encryptionKey field, but
> it is key names, which not aligned with the existing definition of the
> encryptionKey in the binary protocol. Instead of introducing a new one like
> encryptionKeyValue, the proposal will use the existing one(encryptionKey)
> and check the format on the server side. It's not so good, but better than
> adding a new one to confuse users.
> - The proposal quality looks good to me. It provides enough context about
> what is the existing solution and what is the new solution. And provides a
> comprehensive example to show what the new way looks like.
>
> Regards,
> Penghui
>
>
>
> On Mon, Aug 21, 2023 at 5:30 PM Yubiao Feng
>  wrote:
>
> > Sorry, the PR link in the last email is ambiguous,
> > https://github.com/apache/pulsar/pull/20923 is the correct one.
> >
> > Thanks
> > Yubiao Feng
> >
> > On Mon, Aug 21, 2023 at 4:07 PM Yubiao Feng 
> > wrote:
> >
> > > > Hello, Guys
> > > >
> > > > Since there are no concerns in the discussion mail, I'd like to start
> > > > voting for this PIP.
> > > >
> > > > The PIP link: https://github.com/apache/pulsar/pull/
> > > > 20923
> > > > 
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> >

Re: [DISCUSS] New label type: category/*

[mattisonchao]

+1.


Quality and performance are now the most essential parts of Apache Pulsar. 
Adding a clear label to help manage and measure would be great.

Best,
Mattison
On 21 Aug 2023 at 15:30 +0800, PengHui Li , wrote:
> > These tags are clear. Is there any existing document for what the
> labels represent? If not, we'd better create one.
>
> I think no, maybe we can have a page for Committers under the Community tab
>
>
> Regards,
> Penghui
>
>
> > On Mon, Aug 21, 2023 at 2:29 PM Yubiao Feng 
> >  wrote:
> > > @Penghui
> > >
> > > It feels great, and it's really helpful for the community to sift through
> > > highlighting PRs
> > >
> > > Thanks
> > > Yubiao
> > >
> > >
> > >
> > > On Mon, Aug 21, 2023 at 12:28 PM PengHui Li  wrote:
> > >
> > > > Sorry, accidentally clicked send button :), please ignore the previous
> > > > email.
> > > >
> > > > Hi all,
> > > >
> > > > I would like to start a discussion about adding a new label type 
> > > > category/*
> > > > There are three labels are supposed to be added
> > > >
> > > > - category/functionality
> > > > - category/reliability
> > > > - category/performance
> > > >
> > > > It could be a good improvement to the labeling system that Pulsar has
> > > > today.
> > > > Now, Pulsar mainly have four label categories
> > > >
> > > > - type/* - type/bug, type/feat ...
> > > > - component/* - component/authentication, component/broker ...
> > > > - release/* - release/3.0.2, release/3.1.1 ...
> > > > - cherry-picked/* - cherry-picked/branch-3.0, cherry-picked/branch-3.1 
> > > > ...
> > > >
> > > > type/* is mainly distinguish the issues and PRs are for bug reporting, 
> > > > bug
> > > > fix,
> > > > feature requests, feature support.
> > > >
> > > > component/*, you can know the issues and PRs are happened on which
> > > > component with the component labels, such as an issue with type/bug and
> > > > component/broker means it's a bug report for the broker component.
> > > >
> > > > release/* labels are indicating which version the issue/PR has been 
> > > > fixed
> > > > or will be fixed depending on if the version is released or not.
> > > >
> > > > cherry-picked/* labels are more mainly for Pulsar committers to ensure 
> > > > the
> > > > fixes are cherry-picked to the release branches. The label only can be
> > > > added
> > > > after the cherry-picking is done for a corresponding branch. So that the
> > > > release
> > > > manager can have a list of PRs that should to be cherry-picked.
> > > >
> > > > But, In addition to being able to identify which component that the 
> > > > issue,
> > > > PR
> > > > is fixed or enhanced. The category labels will provide more information
> > > > about
> > > > the fix or enhancement for functionality, reliability, or performance. 
> > > > For
> > > > most
> > > > cases, the category labels only work with type/bug and type/enhancement.
> > > >
> > > > - category/functionality,  some functions are not working, such as 
> > > > getting
> > > > errors.
> > > > - category/reliability, the function is working for most cases. It does 
> > > > not
> > > > work
> > > >                                properly in certain specific 
> > > >environments or
> > > > failures. e.g.
> > > >                                data lost, consumption stuck ...
> > > > - category/performance, performance issues fix or improvements.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Mon, Aug 21, 2023 at 12:22 PM PengHui Li  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion about adding a new label type
> > > > category/*
> > > > > There are three labels are supposed to be added
> > > > >
> > > > > - category/functionality
> > > > > - category/reliability
> > > > > - category/performance
> > > > >
> > > > > It could be a good improvement to the labeling system that Pulsar has
> > > > > today.
> > > > > Now, Pulsar mainly have four label categories
> > > > >
> > > > > - type/* - type/bug, type/feat ...
> > > > > - component/* - component/authentication, component/broker ...
> > > > > - release/* - release/3.0.2, release/3.1.1 ...
> > > > > - cherry-picked/* - cherry-picked/branch-3.0, cherry-picked/branch-3.1
> > > > ...
> > > > >
> > > > > type/* is mainly distinguish the issues and PRs are for bug reporting,
> > > > bug
> > > > > fix,
> > > > > feature requests, feature support.
> > > > >
> > > > > component/*, you can know the issues and PRs are happened on which
> > > > > component with the component labels, such as an issue with type/bug 
> > > > > and
> > > > > component/broker means it's a bug report for the broker component.
> > > > >
> > > > > release/* labels are indicating which version the issue/PR has been 
> > > > > fixed
> > > > > or will be fixed depending on if the version is released or not.
> > > > >
> > > > > cherry-picked/* labels are more mainly for Pulsar committers to ensure
> > > > the
> > > > > fixes are cherry-picked to the release branches. The label only can be
> > > > > added
> > 

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

[mattisonchao]

Hi,

I agree with this change to improve the stability of the pulsar cluster.

Just one concern. it's better to move this discussion to a new PIP. because you 
wanna introduce a new broker configuration. 
`brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute`

FYI: https://github.com/apache/pulsar/blob/master/pip/README.md

Looking forward this change and thanks for your contribution. :)

Best,
Mattison



On 7 Jul 2023 at 15:30 +0800, labuladong , wrote:
> Thanks you guys.
>
>
> I agree that per-minute is better than per-second, which is more 
> flexible.
>
>
> I open an issue here:
>
>
> https://github.com/apache/pulsar/issues/20753
>
>
> Regards,
> donglai

[VOTE] PIP-284 Replace reader with table view in the topic policy service

[mattisonchao]

Hello, Guys

Since there are no concerns in the discussion mail, I'd like to start voting 
for this PIP.

The PIP link: https://github.com/apache/pulsar/pull/21033

Best,
Mattison

Re: [VOTE] PIP-293: Delete config `disableBrokerInterceptors`

[mattisonchao]

+1 (binding)


Best,
Mattison
On 15 Aug 2023 at 16:24 +0800, guo jiwei , wrote:
> Hi dev,
> There is a config in ServiceConfiguration called
> `disableBrokerInterceptors` introduced by #8157
> , it looks to disable the
> broker interceptor, but commented for using test only. In actual, we judge
> whether enable the interceptor by judging whether the broker is loaded into
> the interceptors.Then in #10489
> , it kept the same
> implementation.
> But #20422  has changed the
> behavior, it judges whether enable the interceptor by
> `disableBrokerInterceptors` and then it caused an NPE issue mentioned in
> #20710 .
> This config has a little confusing, so we decide to delete it.
>
> PIP: https://github.com/apache/pulsar/pull/20993
>
>
>
> Regards
> Jiwei Guo (Tboy)

Re: [VOTE] Pulsar Release 3.1.0 Candidate 1

[mattisonchao]

+1 (binding)

```
Apache Maven 3.9.3 (21122926829f1ead511c958d89bd2f672198ae9f)
Maven home: /usr/local/Cellar/maven/3.9.3/libexec
Java version: 17.0.7, vendor: Homebrew, runtime: 
/usr/local/Cellar/openjdk@17/17.0.7/libexec/openjdk.jdk/Contents/Home
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "13.4", arch: "x86_64", family: "mac"
```

• Built from the source.
• License Checked
• Ran simple pub/sub by standalone.
• Run simple pub/sub by standalone image.



Best,
Mattison
On 9 Aug 2023 at 09:51 +0800, guo jiwei , wrote:
> This is the first release candidate for Apache Pulsar version 3.1.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar/milestone/35?closed=1
>
> *** Please download, test and vote on this release. This vote will
> stay open for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided
> for convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.0-candidate-1/
>
> SHA-512 checksums:
>
> d3cc4850d31a48f218575e85419e5752e377d2900476bf63381747e307046e9beb5d44b7f45ffd1a14dc971b707824700a0192135e1a63cf0746a3061e261399
>
> apache-pulsar-3.1.0-bin.tar.gz
>
> a97557e65116736ad98b19bbcd81926013eec2591b89237913a96960d08a81f656feb6c8ad1a5a74eaf9b59369b4ec8bf4b5b09a0bd0d6ed072fda986e115901
>
> apache-pulsar-3.1.0-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1241/
>
> The tag to verify:
> v3.1.0-candidate-1 (8d89904bcc9e0a726435fec44e3eb03663924231)
> https://github.com/apache/pulsar/releases/tag/v3.1.0-candidate-1
>
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Docker images:
>
> pulsar images:
> https://hub.docker.com/layers/mattison/pulsar/3.1.0-no-git/images/sha256-157aa5501a5d7fedb843e4e45d808c7276887d89b3bd5ffe22a41bbec9d1da3c?context=explore
>
> pulsar-all images:
> https://hub.docker.com/layers/mattison/pulsar-all/3.1.0-no-git/images/sha256-92837021caca0b68c340c7959e642d52a5f41699293ece7831882afc109558cf?context=explore
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>
>
> Regards
> Jiwei Guo (Tboy)

Re: [VOTE] Accept pulsar-admin-go as part of the Apache Pulsar project

[mattisonchao]

+1 (binding)

Best,
Mattison
On 5 Aug 2023 at 20:05 +0800, Yunze Xu , wrote:
> +1 (binding)
>
> Thanks,
> Yunze
>
> On Sat, Aug 5, 2023 at 12:15 PM tison  wrote:
> >
> > Hi,
> >
> > I'd like to start a vote thread for accepting pulsar-admin-go[1] as part of
> > the Apache Pulsar project. pulsar-admin-go is a Golang client library
> > interacting with Pulsar Admin API. You can see the full proposal, including
> > the motivation and features at [2].
> >
> > Please vote with your opinions. The VOTE will remain open for at least 72
> > hours.
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> > Vote from PMC members is binding.
> >
> > Best,
> > tison.
> >
> > [1] https://github.com/streamnative/pulsar-admin-go
> > [2] https://lists.apache.org/thread/tb6c10qlkg9fj56qc0ldkwc79j7qm0vc

Re: [VERIFY] Pulsar Release 3.1.0 Candidate 1

[mattisonchao]

Hi, Zike

The new candidate docker image has been uploaded. Could you please help review 
it again? Thanks!

pulsar:

https://hub.docker.com/layers/mattison/pulsar/3.1.0-no-git/images/sha256-157aa5501a5d7fedb843e4e45d808c7276887d89b3bd5ffe22a41bbec9d1da3c?context=explore

pulsar-all:

https://hub.docker.com/layers/mattison/pulsar-all/3.1.0-no-git/images/sha256-92837021caca0b68c340c7959e642d52a5f41699293ece7831882afc109558cf?context=explore

The docker image tag is a little bit weird though, but we can retag it when 
moving it to apachepulsar repo.

Best,
Mattison
On 4 Aug 2023 at 11:54 +0800, Zike Yang , wrote:
> Hi, Mattison,
>
> > Note that this RC doesn't require a formal vote, but we would also
> appreciate your feedback with +1/-1. And please provide specific
> comments if your feedback is not +1.
>
> The `-1` is my feedback. It's not the vote. It's up to the release
> manager to decide whether to start a new RC immediately.
> Also, the -1 here doesn't block the verification.
>
> BR,
> Zike Yang
>
> On Fri, Aug 4, 2023 at 11:45 AM  wrote:
> >
> > Hi, Dave
> >
> > I see. Thanks!
> >
> > Best,
> > Mattison
> > On 4 Aug 2023 at 11:18 +0800, Dave Fisher , wrote:
> > > > Hi -
> > > >
> > > > Since this is a VERiFY then the -1 with a technical reason is good data 
> > > > for the Release Manager. They can decide when there is enough feedback 
> > > > before cutting a new release.
> > > >
> > > > Good work!
> > > >
> > > > Best,
> > > > Dave
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > > On Aug 3, 2023, at 7:49 PM, mattisonc...@gmail.com wrote:
> > > > > >
> > > > > > Hi, Zike Yang
> > > > > >
> > > > > > We can rebuild this image to support multiple versions. But I think 
> > > > > > we don't need to start another candidate.
> > > > > >
> > > > > > Also, I wonder if it's ok to give -1 here. We should only set this 
> > > > > > item to required if the new release process has been updated.
> > > > > >
> > > > > > Plus, we should add this part to the build-shell script once it is 
> > > > > > required.[1]
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Mattison
> > > > > >
> > > > > >
> > > > > > [1] https://github.com/apache/pulsar/blob/master/docker/build.sh
> > > > > > > > > > On 3 Aug 2023 at 23:44 +0800, Zike Yang , 
> > > > > > > > > > wrote:
> > > > > > > > > > -1
> > > > > > > > > >
> > > > > > > > > > The docker image doesn't support the `arm64` arch.
> > > > > > > > > > Please use this command to build the image:
> > > > > > > > > > ```shell
> > > > > > > > > > mvn install 
> > > > > > > > > > -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \
> > > > > > > > > > -DskipTests \
> > > > > > > > > > -Pmain,docker -Pdocker-push \
> > > > > > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
> > > > > > > > > > -Ddocker.organization=$DOCKER_USER \
> > > > > > > > > > -pl docker/pulsar,docker/pulsar-all
> > > > > > > > > > ```
> > > > > > > > > >
> > > > > > > > > > The release process of the docker image is still being 
> > > > > > > > > > updated. But
> > > > > > > > > > you can check this PR for more information:
> > > > > > > > > > https://github.com/apache/pulsar-site/pull/595
> > > > > > > > > >
> > > > > > > > > > BR,
> > > > > > > > > > Zike Yang
> > > > > > > > > >
> > > > > > > > > > > > >> On Thu, Aug 3, 2023 at 8:53 PM guo jiwei 
> > > > > > > > > > > > >>  wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> This is the first release candidate for Apache 
> > > > > > > > > > > > >> Pulsar version 3.1.0.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> It fixes the following issues:
> > > > > > > > > > > > >> https://github.com/apache/pulsar/milestone/35?closed=1
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> *** Please download, test and verify on this 
> > > > > > > > > > > > >> release. This release
> > > > > > > > > > > > >> candidate verification will stay open until Aug 8 ***
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Note that we are verifying upon the source (tag), 
> > > > > > > > > > > > >> binaries are provided for
> > > > > > > > > > > > >> convenience.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Source and binary files:
> > > > > > > > > > > > >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.0-candidate-1/
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> SHA-512 checksums:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> d3cc4850d31a48f218575e85419e5752e377d2900476bf63381747e307046e9beb5d44b7f45ffd1a14dc971b707824700a0192135e1a63cf0746a3061e261399
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> apache-pulsar-3.1.0-bin.tar.gz
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> a97557e65116736ad98b19bbcd81926013eec2591b89237913a96960d08a81f656feb6c8ad1a5a74eaf9b59369b4ec8bf4b5b09a0bd0d6ed072fda986e115901
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> apache-pulsar-3.1.0-src.tar.gz
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Maven staging repo:
> > > > > > 

Re: [VERIFY] Pulsar Release 3.1.0 Candidate 1

[mattisonchao]

Hi, Dave

I see. Thanks!

Best,
Mattison
On 4 Aug 2023 at 11:18 +0800, Dave Fisher , wrote:
> Hi -
>
> Since this is a VERiFY then the -1 with a technical reason is good data for 
> the Release Manager. They can decide when there is enough feedback before 
> cutting a new release.
>
> Good work!
>
> Best,
> Dave
>
> Sent from my iPhone
>
> > On Aug 3, 2023, at 7:49 PM, mattisonc...@gmail.com wrote:
> >
> > Hi, Zike Yang
> >
> > We can rebuild this image to support multiple versions. But I think we 
> > don't need to start another candidate.
> >
> > Also, I wonder if it's ok to give -1 here. We should only set this item to 
> > required if the new release process has been updated.
> >
> > Plus, we should add this part to the build-shell script once it is 
> > required.[1]
> >
> > WDYT?
> >
> > Best,
> > Mattison
> >
> >
> > [1] https://github.com/apache/pulsar/blob/master/docker/build.sh
> > > > On 3 Aug 2023 at 23:44 +0800, Zike Yang , wrote:
> > > > -1
> > > >
> > > > The docker image doesn't support the `arm64` arch.
> > > > Please use this command to build the image:
> > > > ```shell
> > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \
> > > > -DskipTests \
> > > > -Pmain,docker -Pdocker-push \
> > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
> > > > -Ddocker.organization=$DOCKER_USER \
> > > > -pl docker/pulsar,docker/pulsar-all
> > > > ```
> > > >
> > > > The release process of the docker image is still being updated. But
> > > > you can check this PR for more information:
> > > > https://github.com/apache/pulsar-site/pull/595
> > > >
> > > > BR,
> > > > Zike Yang
> > > >
> > > > >> On Thu, Aug 3, 2023 at 8:53 PM guo jiwei  
> > > > >> wrote:
> > > > >>
> > > > >> This is the first release candidate for Apache Pulsar version 3.1.0.
> > > > >>
> > > > >> It fixes the following issues:
> > > > >> https://github.com/apache/pulsar/milestone/35?closed=1
> > > > >>
> > > > >> *** Please download, test and verify on this release. This release
> > > > >> candidate verification will stay open until Aug 8 ***
> > > > >>
> > > > >> Note that we are verifying upon the source (tag), binaries are 
> > > > >> provided for
> > > > >> convenience.
> > > > >>
> > > > >> Source and binary files:
> > > > >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.0-candidate-1/
> > > > >>
> > > > >> SHA-512 checksums:
> > > > >>
> > > > >> d3cc4850d31a48f218575e85419e5752e377d2900476bf63381747e307046e9beb5d44b7f45ffd1a14dc971b707824700a0192135e1a63cf0746a3061e261399
> > > > >>
> > > > >> apache-pulsar-3.1.0-bin.tar.gz
> > > > >>
> > > > >> a97557e65116736ad98b19bbcd81926013eec2591b89237913a96960d08a81f656feb6c8ad1a5a74eaf9b59369b4ec8bf4b5b09a0bd0d6ed072fda986e115901
> > > > >>
> > > > >> apache-pulsar-3.1.0-src.tar.gz
> > > > >>
> > > > >> Maven staging repo:
> > > > >> https://repository.apache.org/content/repositories/orgapachepulsar-1241/
> > > > >>
> > > > >> The tag to verify:
> > > > >> v3.1.0-candidate-1 (8d89904bcc9e0a726435fec44e3eb03663924231)
> > > > >> https://github.com/apache/pulsar/releases/tag/v3.1.0-candidate-1
> > > > >>
> > > > >> Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > >> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > > > >>
> > > > >> Docker images:
> > > > >>
> > > > >> pulsar images:
> > > > >> https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore
> > > > >>
> > > > >> pulsar-all images:
> > > > >> https://hub.docker.com/layers/mattison/pulsar-all/3.1.0-candidate-1/images/sha256-d12f9a1af99979b9a20babdefebc8645f737318d879b77eb06f171489269e1fc?context=explore
> > > > >>
> > > > >> Please download the source package, and follow the README to build
> > > > >> and run the Pulsar standalone service.
> > > > >>
> > > > >> Note that this RC doesn't require a formal vote, but we would also
> > > > >> appreciate your feedback with +1/-1. And please provide specific
> > > > >> comments if your feedback is not +1.
> > > > >>
> > > > >> Regards
> > > > >> Jiwei Guo (Tboy)
>

Re: [VERIFY] Pulsar Release 3.1.0 Candidate 1

[mattisonchao]

Hi, Zike Yang

We can rebuild this image to support multiple versions. But I think we don't 
need to start another candidate.

Also, I wonder if it's ok to give -1 here. We should only set this item to 
required if the new release process has been updated.

Plus, we should add this part to the build-shell script once it is required.[1]

WDYT?

Best,
Mattison


[1] https://github.com/apache/pulsar/blob/master/docker/build.sh
On 3 Aug 2023 at 23:44 +0800, Zike Yang , wrote:
> -1
>
> The docker image doesn't support the `arm64` arch.
> Please use this command to build the image:
> ```shell
> mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \
> -DskipTests \
> -Pmain,docker -Pdocker-push \
> -Ddocker.platforms=linux/amd64,linux/arm64 \
> -Ddocker.organization=$DOCKER_USER \
> -pl docker/pulsar,docker/pulsar-all
> ```
>
> The release process of the docker image is still being updated. But
> you can check this PR for more information:
> https://github.com/apache/pulsar-site/pull/595
>
> BR,
> Zike Yang
>
> On Thu, Aug 3, 2023 at 8:53 PM guo jiwei  wrote:
> >
> > This is the first release candidate for Apache Pulsar version 3.1.0.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar/milestone/35?closed=1
> >
> > *** Please download, test and verify on this release. This release
> > candidate verification will stay open until Aug 8 ***
> >
> > Note that we are verifying upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.1.0-candidate-1/
> >
> > SHA-512 checksums:
> >
> > d3cc4850d31a48f218575e85419e5752e377d2900476bf63381747e307046e9beb5d44b7f45ffd1a14dc971b707824700a0192135e1a63cf0746a3061e261399
> >
> > apache-pulsar-3.1.0-bin.tar.gz
> >
> > a97557e65116736ad98b19bbcd81926013eec2591b89237913a96960d08a81f656feb6c8ad1a5a74eaf9b59369b4ec8bf4b5b09a0bd0d6ed072fda986e115901
> >
> > apache-pulsar-3.1.0-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1241/
> >
> > The tag to verify:
> > v3.1.0-candidate-1 (8d89904bcc9e0a726435fec44e3eb03663924231)
> > https://github.com/apache/pulsar/releases/tag/v3.1.0-candidate-1
> >
> > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Docker images:
> >
> > pulsar images:
> > https://hub.docker.com/layers/mattison/pulsar/3.1.0-candidate-1/images/sha256-0efbaad7d893cc5041a46a2d4d56432bda855ae4068a38349777d1be6e98d27d?context=explore
> >
> > pulsar-all images:
> > https://hub.docker.com/layers/mattison/pulsar-all/3.1.0-candidate-1/images/sha256-d12f9a1af99979b9a20babdefebc8645f737318d879b77eb06f171489269e1fc?context=explore
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> > Note that this RC doesn't require a formal vote, but we would also
> > appreciate your feedback with +1/-1. And please provide specific
> > comments if your feedback is not +1.
> >
> > Regards
> > Jiwei Guo (Tboy)

Re: [VOTE] Pulsar Release 3.0.1 Candidate 2

[mattisonchao]

+1 (binding)

• Built from source with JDK 17.0.7 and Maven 3.9.3 on Intel MacOS
• Ran produce/consume test
• Ran license check
• Ran produce/consume test based on docker server


Best,
Mattison
On 4 Aug 2023 at 02:24 +0800, Yunze Xu , wrote:
> +1 (binding)
>
> - Verified the signatures and checksum
> - Build from source with JDK 17.0.7 and Maven 3.9.3 on m1 macOS (aarch)
> - Test basic end to end cases
> - Test it with KoP 3.0.0.4
>
> Thanks,
> Yunze
>
> On Thu, Aug 3, 2023 at 7:43 PM PengHui Li  wrote:
> >
> > +1(binding)
> >
> > ```
> > openjdk version "17.0.7" 2023-04-18 LTS
> > Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
> > OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: "mac"
> > ```
> >
> > - Checked the signature
> > - Run the perf test
> > - Build from source
> > - Run the Cassandra connector validation
> > - Run the stateful function validation
> >
> > Thanks,
> > Penghui
> >
> > On Sun, Jul 30, 2023 at 5:31 PM guo jiwei  wrote:
> >
> > > > +1 (binding)
> > > >
> > > > - Checked the signature
> > > > - Build from source (maven 3.8.6, JDK 17.0.3)
> > > > - Start standalone
> > > > - Publish and Consume
> > > > - Verified Cassandra connector
> > > > - Verified stateful function
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > > >
> > > >
> > > > On Wed, Jul 26, 2023 at 6:05 PM Zike Yang  wrote:
> > > >
> > > > > > This is the second release candidate for Apache Pulsar, version 
> > > > > > 3.0.1.
> > > > > >
> > > > > > It fixes the following issues:
> > > > > > https://github.com/apache/pulsar/compare/v3.0.0...v3.0.1-candidate-2
> > > > > >
> > > > > > *** Please download, test and vote on this release. This vote will 
> > > > > > stay
> > > > > > open
> > > > > > for at least 72 hours ***
> > > > > >
> > > > > > Note that we are voting upon the source (tag), binaries are 
> > > > > > provided for
> > > > > > convenience.
> > > > > >
> > > > > > Source and binary files:
> > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-3.0.1-candidate-2/
> > > > > >
> > > > > > SHA-512 checksums:
> > > > > >
> > > > > >
> > > > > >
> > > > b8d8c0afe9923dbf1c3adfdbdabceb8d336f2cb0fe99d6057913f2892b34695ee266de6554b6d50ab49af54204fac1de6a67ee2b20bc2127dbade7034bf669f5
> > > > > > ./apache-pulsar-3.0.1-bin.tar.gz
> > > > > >
> > > > > >
> > > > > >
> > > > 6c6561b5b59a3e0bdb527980130979f3394930ff55b3cd3e6391ad127fd6cd11fc544b60736132917164ef8a8d0a4c9e52d1de01b3ffcc6b1e10ce337a217553
> > > > > > ./apache-pulsar-3.0.1-src.tar.gz
> > > > > >
> > > > > > Maven staging repo:
> > > > > > https://repository.apache.org/content/repositories/orgapachepulsar-1240/
> > > > > >
> > > > > > The tag to be voted upon:
> > > > > > v3.0.1-candidate-2 (d32b2bc6f031e9a8e0fbf7f8212dc6d56df2deb2)
> > > > > > https://github.com/apache/pulsar/commits/v3.0.1-candidate-2
> > > > > >
> > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > > > > >
> > > > > > Docker images:
> > > > > >
> > > > > > pulsar:
> > > > > >
> > > > https://hub.docker.com/layers/snzkyang/pulsar/3.0.1-d32b2bc/images/sha256-05f2005f1f988250536a7fd160f94cabef4b850d82ac132fea6e6151824104a1?context=repo
> > > > > >
> > > > > > pulsar-all:
> > > > > >
> > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.1-d32b2bc/images/sha256-40de86dccb1a76f4b6b011be94d7c36e49c18a0a42560363c0b6dd3fd2935d7a?context=repo
> > > > > >
> > > > > > Please download the source package, and follow the README to build
> > > > > > and run the Pulsar standalone service.
> > > > > >
> > > >

Re: [DISCUSS] Support key filter for Table View

[mattisonchao]

I've extended the interface to allow filtering by key and value.

Thanks
Mattison
On 18 Jul 2023 at 09:50 +0800, mattisonc...@gmail.com, wrote:
>
> Hi, Folks
>
> I would like to follow the PIP process to start the discussion of PIP-283.
>
> Please don't hesitate to leave your concern and suggestions.
>
> Best,
> Mattison

[DISCUSS] PIP-284 Refactor SystemTopicBasedTopicPoliciesService by TableView

[mattisonchao]

Hi, Folks

I would like to follow the PIP process to start the discussion of PIP-284[1].

Please don't hesitate to leave your concern and suggestions.

Best,
Mattison

[1] https://github.com/apache/pulsar/pull/20828

Re: [DISCUSS] Support key filter for Table View

[mattisonchao]

Sorry, I wrote this email through another application. It seems the link does 
not support well in the apache pony mail.

I'll put the link over here.

PIP-283: https://github.com/apache/pulsar/pull/20827

Best,
Mattison
On 18 Jul 2023 at 09:50 +0800, mattisonc...@gmail.com, wrote:
>
> Hi, Folks
>
> I would like to follow the PIP process to start the discussion of PIP-283.
>
> Please don't hesitate to leave your concern and suggestions.
>
> Best,
> Mattison

[DISCUSS] Support key filter for Table View

[mattisonchao]

Hi, Folks

I would like to follow the PIP process to start the discussion of PIP-283.

Please don't hesitate to leave your concern and suggestions.

Best,
Mattison

Re: [VOTE] Pulsar Release 2.11.2 Candidate-1

[mattisonchao]

+1 (binding)

• Built from source. ( maven 3.9.3, JDK 17.0.7 )
• Ran standalone with basic publishing & consuming.
• Passed binary license check.
• Ran pulsar-all docker image with basic publishing & consuming.


Best,
Mattison
On 16 Jul 2023 at 10:39 +0800, guo jiwei , wrote:
> +1 (binding)
>
> - Checked the signature
> - Build from source
> - Start standalone
> - Publish and Consume
> - Verified Cassandra connector
> - Verified stateful function
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Fri, Jul 14, 2023 at 6:24 PM PengHui Li  wrote:
>
> > +1 (binding)
> >
> > Environment:
> >
> > openjdk version "17.0.7" 2023-04-18 LTS
> > Apache Maven 3.8.8
> >
> > - Checked the signature
> > - Build from source
> > - Start standalone
> > - Publish and Consume
> > - Verified Cassandra connector
> > - Verified stateful function
> >
> > Regards,
> > Penghui
> >
> > On Thu, Jul 13, 2023 at 12:03 PM guo jiwei  wrote:
> >
> > > > This is the first release candidate for Apache Pulsar, version 2.11.2.
> > > >
> > > > This release contains 123 commits by 47 contributors.
> > > > https://github.com/apache/pulsar/compare/v2.11.1...v2.11.2-candidate-1
> > > >
> > > > CI for this release candidate
> > > > https://github.com/Technoboy-/pulsar/pull/40
> > > >
> > > > *** Please download, test and vote on this release. This vote will stay
> > > > open
> > > > for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.2-candidate-1
> > > >
> > > > SHA-512 checksums:
> > > >
> > > >
> > > >
> > 21f68ae8d651d2369557a80d65cca0a10eed3e72e146b51adb8107d369d57e31ba8129de7f4a343b875397d63584f1ec535ec48fd6aafc0697e33ea5118ce3e0
> > > >
> > > > ./apache-pulsar-2.11.2-bin.tar.gz
> > > >
> > > >
> > > >
> > a8adb23eb06c564236136272fc2bd0c52e8a1ceb731bf863b9719f08f2ad44982f26763378dc3e9b6418d80a836b0fb4a6289a93d98ef703391d78a1114ba086
> > > >
> > > >
> > > > ./apache-pulsar-2.11.2-src.tar.gz
> > > >
> > > > Maven staging repo:
> > > > https://repository.apache.org/content/repositories/orgapachepulsar-1233/
> > > >
> > > > The tag to be voted upon:
> > > > v2.11.2-candidate-1 (ceed96f7af848120acb594ced7a19f9a727e640a)
> > > > https://github.com/apache/pulsar/releases/tag/v2.11.2-candidate-1
> > > >
> > > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > > >
> > > >
> > > >
> > https://registry.hub.docker.com/layers/mattison/pulsar-all/2.11.2/images/sha256-fdcb80f2e4d419c044258c3513cb5777f247a03588932e7609ce733c196444f5
> > > >
> > > >
> > > >
> > https://registry.hub.docker.com/layers/mattison/pulsar/2.11.2/images/sha256-199b32ed8aa6174d2e3fcd50f5c5690914b23f5bb56f68a959e2173376377f27
> > > >
> > > > Please download the source package, and follow the
> > > > release-candidate-validation doc to build
> > > > and run the Pulsar standalone service.
> > > > https://pulsar.apache.org/contribute/validate-release-candidate
> > > >
> > > > Since the metadata store is changed from ZK to RocksDB, the
> > > > verification of the `stateful functions` needs to set the parameter
> > > > "export PULSAR_STANDALONE_USE_ZOOKEEPER=1"
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > > >
> >

RE: Re: [VOTE] PIP-280 : Refactor CLI Argument Parsing Logic for Measurement Units using JCommander's custom converter

[mattisonchao]

+1(binding)

Mattison
On 10 Jul 2023 at 19:55 +0800, Joo Hyuk Kim , wrote:
> Thanks,
> JooHyuk.
>
> On 2023/07/10 00:24:20 Zili Chen wrote:
> > +1 (non-binding)
> >
> > On 2023/07/07 09:25:22 Joo Hyuk Kim wrote:
> > > > Hi community,
> > > >
> > > > This PIP has received a couple of approvals in github PR link [1]
> > > > So I thought it's time to vote.
> > > >
> > > > ## Motivation
> > > >
> > > > In the current Pulsar codebase, the logic to parse CLI arguments for
> > > > measurement units like time and bytes is
> > > >
> > > > scattered across various CLI classes. Each value read has its distinct
> > > > parsing implementation, leading to a lack of code
> > > >
> > > > reuse.
> > > >
> > > >
> > > > ## Goals
> > > >
> > > >
> > > > This PIP is to refactor the argument parsing logic to leverage the
> > > > `@Parameter.converter`
> > > >
> > > > functionality provided by JCommander [link 3]. This will isolate the
> > > > measurement-specific parsing logic and increase
> > > >
> > > > code
> > > >
> > > > reusability.
> > > >
> > > >
> > > > ### In Scope
> > > >
> > > >
> > > > - Refactor all `Cmd` classes to utilize the converter functionality of
> > > > JCommander. This will streamline the parsing
> > > >
> > > > logic and simplify the codebase.
> > > >
> > > > - Refer to bottom section "Concrete Example", before "Links"
> > > >
> > > > - Or on-going PR with small use case in
> > > > https://github.com/apache/pulsar/pull/20663
> > > >
> > > >
> > > > ## links
> > > >
> > > >
> > > > [1] PR : https://github.com/apache/pulsar/pull/20691
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > JooHyukKim (Vince)
> > > >
> >

Re: [VOTE] Pulsar Client Go Release 0.11.0 Candidate 1

[mattisonchao]

+1 (binding)

• Downloaded the source code.
• Built from the source code on macOS.
• Ran make test.
• Ran all of the examples with pulsar-3.0.0.


Best,
Mattison
On 10 Jul 2023 at 11:37 +0800, Rui Fu , wrote:
> +1
>
> - verified checksum and signature
> - run producer and consumer examples
>
> Best,
>
> Rui Fu
> On Jul 4, 2023 at 17:22 +0800, Zike Yang , wrote:
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version
> > 0.11.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This is the first release candidate for Apache Pulsar Go client, version 
> > 0.11.0.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar-client-go/milestone/13?closed=1
> >
> > Pulsar Client Go's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download these packages and review this release candidate:
> > - Review release notes: https://github.com/apache/pulsar-client-go/pull/1048
> > - Download the source package (verify shasum, and asc) and follow the
> > README.md to build and run the pulsar-client-go.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Source file:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.0-candidate-1/
> >
> > The tag to be voted upon:
> > v0.11.0
> > https://github.com/apache/pulsar-client-go/tree/v0.11.0-candidate-1
> >
> > SHA-512 checksums:
> > 72cf0f71c82095f0d412ff15b35e32c56f118ec7f297ad662073c14786fead0a416d7b1a8b8caa890b76bd722cba1734e52fd4f53ee3daa793cb5a7915e5fb32
> > apache-pulsar-client-go-0.11.0-src.tar.gz

Re: [DISCUSS] About cherry-picking `Use Ubuntu 22.04 for Pulsar images (#20475)`

[mattisonchao]

Agree with cherry-pick PR.

Best
Mattison
On 26 Jun 2023 at 17:10 +0800, guo jiwei , wrote:
> Hello community:
> For this patch Use Ubuntu 22.04 for Pulsar images
> , we need some ideas about
> cherry-picking to other branches.
>
>
> Regards
> Jiwei Guo (Tboy)

Re: [DISCUSS] Make system topics non-partitioned by default

[mattisonchao]

Hi, Michael

I've got several questions and hope you can help answer.

Why not disable auto-creation for system topics? and then we can create it 
before the broker start. which can avoid introduce complex meaning of 
`allowAutoTopicCreationType` configuration.

Plus, why we chose the non-partitioned topic? maybe we can just create one 
partition? because the non-partitioned topic can't scale by updating the 
partition number.

Best,
Mattison
On 28 Jun 2023 at 06:45 +0800, Michael Marshall , wrote:
> Hi Pulsar Community,
>
> After several recent issues regarding incorrectly auto created topics,
> I think we should default to creating system topics as non-partitioned
> topics when auto created.
>
> Lari put together a PR here
> https://github.com/apache/pulsar/pull/20397. The PR has relevant
> context.
>
> Let me know what you think.
>
> Thanks,
> Michael

Re: [VOTE] PIP-278: Support pluggable topic compaction service

[mattisonchao]

+1 (binding)

Best,
Mattison
On 26 Jun 2023 at 10:01 +0800, Cong Zhao , wrote:
> Hello, community:
>
> This thread is to start a vote for PIP-278: Support pluggable topic
> compaction service.
>
> Discussion thread:
> https://lists.apache.org/thread/ox2bot3p9j9fydqkw3v5gt5twc8jslvd
>
> PIP PR: https://github.com/apache/pulsar/pull/20624
>
> Sincerely
> Cong Zhao

Re: [VOTE] PIP-276: Add metric `pulsar_topic_load_times

[mattisonchao]

+1(binding)

Best,
Mattison
On 20 Jun 2023 at 10:45 +0800, PengHui Li , wrote:
> +1 (binding)
>
> Thanks,
> Penghui
>
> On Tue, Jun 20, 2023 at 10:40 AM Yubiao Feng
>  wrote:
>
> > Voting +1 (non-binding)
> >
> > Thanks
> > Yubiao Feng
> >
> > On Mon, Jun 19, 2023 at 5:21 PM Asaf Mesika  wrote:
> >
> > > > Voting +1 (non-binding)
> > > >
> > > > On Fri, Jun 16, 2023 at 12:23 PM guo jiwei  wrote:
> > > >
> > > > > > @Asaf Thanks, I have addressed the comment.
> > > > > >
> > > > > > Regards
> > > > > > Jiwei Guo (Tboy)
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 16, 2023 at 3:55 AM Asaf Mesika 
> > > > wrote:
> > > > > >
> > > > > > > > -1 (non-binding)
> > > > > > > >
> > > > > > > > I'm perfectly ok with the idea; just please fix the document. It
> > looks
> > > > > > too
> > > > > > > > messy. Even 1 paragraph changes can look neat and clean.
> > > > > > > > I left notes in the draft PR you opened for the pip.
> > > > > > > >
> > > > > > > > I'll change my non-binding vote once that's done.
> > > > > > > >
> > > > > > > > On Thu, Jun 15, 2023 at 11:07 AM guo jiwei 
> > > > > > > > 
> > > > wrote:
> > > > > > > >
> > > > > > > > > > Hi, community:
> > > > > > > > > > The metrics are all started with `pulsar_`, so that both 
> > > > > > > > > > users
> > > > and
> > > > > > > > > > operators can quickly find the metrics of the entire system 
> > > > > > > > > > through
> > > > > > > > > > this prefix. However, due to some other reasons, it was 
> > > > > > > > > > found that
> > > > > > > > > > `topic_load_times` was missing the prefix, so want to get 
> > > > > > > > > > it right.
> > > > > > > > > > In the master branch :
> > > > > > > > > > * `pulsar_topic_load_times`: Add this new metric which has 
> > > > > > > > > > the
> > > > > > same
> > > > > > > > > > meaning as `topic_load_times`
> > > > > > > > > > * `topic_load_times`: Mark this metric as deprecated and
> > remove
> > > > > > it
> > > > > > > > in
> > > > > > > > > > the next version
> > > > > > > > > >
> > > > > > > > > > PIP: https://github.com/apache/pulsar/pull/20518
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Jiwei Guo (Tboy)
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >

Re: [VOTE] Pulsar Client Python Release 3.2.0 Candidate 1

[mattisonchao]

+1 (binding)

- Start the standalone (master)
- Start consumer (python3 ./examples/consumer.py)
- Start producer (python3 ./examples/producer.py )

Best,
Mattison
On Jun 2, 2023, 23:09 +0800, Matteo Merli , wrote:
> +1
>
> * Checked signature
> * Installed and tested with Mac & linux wheels
>
>
> Matteo
> --
> Matteo Merli
> 
>
>
> On Fri, Jun 2, 2023 at 7:36 AM PengHui Li  wrote:
>
> > +1 (binding)
> >
> > - Checked the signature
> > - Install pulsar_client-3.2.0-cp311-cp311-macosx_10_15_universal2.whl
> > - Start standalone (master)
> > - Start consumer (python3 ./examples/consumer.py)
> > - Start producer (python3 ./examples/producer.py)
> > - The consumer can receive all the published messages by producer
> >
> > Regards,
> > Penghui
> >
> >
> > On Fri, Jun 2, 2023 at 10:11 PM guo jiwei  wrote:
> >
> > > > +1 (binding)
> > > >
> > > > - Checked the signature
> > > > - Install the python .whl file (pip3 install
> > > > pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl) on macOS
> > > > - Start the standalone (master)
> > > > - Start consumer (python3 ./examples/consumer.py)
> > > > - Start producer (python3 ./examples/producer.py )
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > > >
> > > >
> > > > On Tue, May 30, 2023 at 10:48 AM Yunze Xu  wrote:
> > > >
> > > > > > This is the 1st release candidate for Apache Pulsar Client Python,
> > > > > > version 3.2.0.
> > > > > >
> > > > > > It fixes the following issues:
> > > > > > https://github.com/apache/pulsar-client-python/milestone/3?closed=1
> > > > > >
> > > > > > *** Please download, test and vote on this release. This vote will
> > > > > > stay open for at least 72 hours ***
> > > > > >
> > > > > > Python wheels:
> > > > > >
> > > > > >
> > > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.2.0-candidate-1/
> > > > > >
> > > > > > The supported python versions are 3.7, 3.8, 3.9, 3.10 and 3.11. The
> > > > > > supported platforms and architectures are:
> > > > > > - Windows x86_64 (windows/)
> > > > > > - glibc-based Linux x86_64 (linux-glibc-x86_64/)
> > > > > > - glibc-based Linux arm64 (linux-glibc-arm64/)
> > > > > > - musl-based Linux x86_64 (linux-musl-x86_64/)
> > > > > > - musl-based Linux arm64 (linux-musl-arm64/)
> > > > > > - macOS universal 2 (macos/)
> > > > > >
> > > > > > You can download the wheel (the `.whl` file) according to your own 
> > > > > > OS
> > > > > > and Python version
> > > > > > and install the wheel:
> > > > > > - Windows: `py -m pip install *.whl --force-reinstall`
> > > > > > - Linux or macOS: `python3 -m pip install *.whl --force-reinstall`
> > > > > >
> > > > > > The tag to be voted upon: v3.2.0-candidate-1
> > > > > > (ba99cd5f372ab645bbb12239dad53668b10075a9)
> > > > > >
> > > > > >
> > > >
> > https://github.com/apache/pulsar-client-python/releases/tag/v3.2.0-candidate-1
> > > > > >
> > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > > > https://downloads.apache.org/pulsar/KEYS
> > > > > >
> > > > > > Please download the Python wheels and follow the README to test.
> > > > > >
> > > >
> >

Re: [VOTE] PIP-265: PR-based system for managing and reviewing PIPs

[mattisonchao]

+1 (binding)

Best,
Mattison
On May 15, 2023, 15:38 +0800, Enrico Olivelli , wrote:
> +1 (binding)
>
> The proposal makes sense to me.
> Let's try
>
> Thanks for carrying on with this initative
> Enrico
>
> Il giorno lun 15 mag 2023 alle ore 05:23 Max Xu  ha 
> scritto:
> >
> > +1 (non-binding)
> >
> > Thanks for driving this helpful proposal!
> >
> > Best,
> > Max Xu
> >
> >
> > On Wed, May 10, 2023 at 6:52 PM Asaf Mesika  wrote:
> >
> > > > Hi,
> > > >
> > > > I'm starting the vote process for PIP-265.
> > > >
> > > > Link: https://github.com/apache/pulsar/issues/20207
> > > >
> > > > Thanks!
> > > >
> > > > Asaf
> > > >

Re: [VOTE] Pulsar Client C++ Release 3.2.0 Candidate 3

[mattisonchao]

I found an issue here. https://github.com/apache/pulsar-client-cpp/issues/268
Could you help confirm if it is a problem?

+1 (binding)

• Build from source code on MacOS (12.5) with (Intel Core i7)
• Ran pulsar-tests under the tests/ dir


Best,
Mattison
On May 15, 2023, 18:43 +0800, PengHui Li , wrote:
> +1 (binding)
>
> - Checked the signature
> - Build from source codes on MacOS (13.3.1) with (M2 Pro)
> - Ran pulsar-tests under the tests/ dir
>
> Regards,
> Penghui
>
> On Thu, May 11, 2023 at 6:43 PM Nozomi Kurihara  wrote:
>
> > +1 (binding)
> >
> > - verified signature and checksum
> > - built from source codes on CentOS Linux 7
> > - ran producer and consumer on CentOS Linux 7
> >
> > Thanks,
> > Nozomi
> >

[DISCUSS] Is PIP required for small changes in metrics

[mattisonchao]

Hi, All

I've got two questions want to discuss with you guys.

1. I am wondering if we should draft PIP for small metrics changes, e.g: 
https://github.com/apache/pulsar/pull/20147
2. We haven't declear we should draft a PIP for configuration changes. why?  
Refer to: https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md


I want to discuss with you all to come to a conclusion within the community, 
thanks a lot!

Best,
Mattison

[DISCUSS] Sorting out pulsar's internal thread pools

[mattisonchao]

Hello, folks.

I would like to start discussing the pulsar internal thread pool sorting out.

How did I get this idea?

Recently, we met some problems with the BK operation timeout. After 
investigating, we found an issue that is we share the IO executor(workgroup) 
with the Bookkeeper client and internal client and do some other async task in 
the dispatcher or somewhere to avoid deadlock.

But the problem over here. If we use this executor to do some kind of 
`blocking`(or spend much time computing. e.g. reply to many delayed messages) 
operation, it will block BK clients from sending requests if they are using the 
same thread.

And then, I checked all the usage of the thread pool. We need the rule to 
constrain what thread pool we should use.

What am I expecting?

I want to collect all the thread pools and define a clear usage guide to avoid 
wrong use and improve the fault tolerance(the component problem shouldn't 
affect the whole broker)



I need to hear your guy's opinions. Please feel free to leave any questions. 
Thanks!


Best,
Mattison

Re: [VOTE] Pulsar Release 2.9.5 Candidate 3

[mattisonchao]

+1 (Binding)

 • Built from the source package (maven 3.8.6 OpenJDK 11.0)
 • Ran binary package standalone with pub/sub
 • Ran docker image(pulsar-all) standalone with pub/sub
 • Ran License check

Best,
Mattison
On Apr 10, 2023, 15:38 +0800, Cong Zhao , wrote:
> This is the third release candidate for Apache Pulsar, version 2.9.5.
>
> This release contains 105 commits by 32 contributors.
> https://github.com/apache/pulsar/compare/v2.9.4...v2.9.5-candidate-3
>
> *** Please download, test, and vote on this release. This vote will stay
> open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.5-candidate-3/
>
> SHA-512 checksums:
> c5f1b06f2f7249616a07b732daa036546591c1aa9e9f37e88ddce6d3eaf3d78b2c6da548ef42946d20d70f9397cd5b55ea4e01fe5cb4aa96fb4608bd62635f67
> apache-pulsar-2.9.5-bin.tar.gz
>
> 72dee0fb642a269c5d0aedfa3e56aee503f56af319b67e4628f39da8cdd44f0bbdcd019e121eafca32b7b7665959770a3d91f0618d157837808da9511e9011a40f
> apache-pulsar-2.9.5-src.tar.gz
>
> 738478bbcf323080487f5645b4c1edb9f45d126a8a2cd7e8c5678c8569ec7b21042ecb6fdd4796b65c6a24bd1e5ad082f5c0d15eb1a64408be4f4c53a06d3ef660
> apache-pulsar-offloaders-2.9.5-bin.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1223/
>
> The tag to be voted upon:
> v2.9.5-candidate-3 (7337470f33586aa639854266efe95c2fa32f48db)
> https://github.com/apache/pulsar/releases/tag/v2.9.5-candidate-3
>
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
>
> Docker images:
>
> 
> https://hub.docker.com/layers/czcoder/pulsar/2.9.5/images/sha256-9bcbad78a65a09b3fe5d3acd158574ad0eda4531cc8c49db16760a2ed1364070?context=explore
>
> 
> https://hub.docker.com/layers/czcoder/pulsar-all/2.9.5/images/sha256-94e50aa1d5b91be9698c959c787eaff0677bb3c65dd1a578b1a641fa5a1d1715?context=explore
>
> 
> https://hub.docker.com/layers/czcoder/pulsar-grafana/2.9.5/images/sha256-d857951e85b41fe7172769380ffbcac126c75dc476fdcb4755623600b3eb03e0?context=explore
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.
>
> Thanks
> Cong Zhao

Re: [VOTE] Pulsar Release 2.10.4 Candidate 4

[mattisonchao]

+1 (Binding)

 • Built from the source package (maven 3.8.6 OpenJDK 11)
 • Ran binary package standalone with pub/sub
 • Ran docker image(pulsar-all) standalone with pub/sub
 • Ran License check

Best,
Mattison
On Apr 12, 2023, 17:10 +0800, Xiangying Meng , wrote:
> This is the fourth release candidate for Apache Pulsar, version 2.10.4.
>
> This release contains 126 commits by 37 contributors.
> https://github.com/apache/pulsar/compare/v2.10.3...v2.10.4-candidate-4
>
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.4-candidate-4/
>
> SHA-512 checksums:
> 63343005235be32e970574c9733f06cb472adfdd6511d53b91902d66c805b21cee4039b51b69013bf0f9cbcde82f4cd944c069a7d119d1c908a40716ff82eca3
> apache-pulsar-2.10.4-bin.tar.gz
> 2d3398a758917bccefa8550f3f69ec8a72a29f541bcd45963e6fddaec024cc690b33f1d49392dc2437e332e90a89e47334925a50960c5f8960e34c1ac8ed2543
> apache-pulsar-2.10.4-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1226
>
> The tag to be voted upon:
> v2.10.4-candidate-4
> (1fe05d5cd3ec9f70cd7179efa4b69eac72fd88bd)
> https://github.com/apache/pulsar/releases/tag/v2.10.4-candidate-4
>
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://downloads.apache.org/pulsar/KEYS
>
> Docker images:
>
> 
> https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.4/images/sha256-8b76d49401d3fe398be3cde395fb164ad8722b64691e31c44991f32746ca8119?context=repo
>
> 
> https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.4/images/sha256-c20a13ed215e4837f95a99cf84914d03f557204d44ed610dfc41d2e23a77a92c?context=repo
>
> Please download the source package, and follow the README to build
> and run the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.11.1 Candidate-2

[mattisonchao]

+1 (Binding)

 • Built from the source package (maven 3.8.6 OpenJDK 17.0)
 • Ran binary package standalone with pub/sub
 • Ran docker image(pulsar-all) standalone with pub/sub
 • Ran License check

Best,
Mattison
On Apr 13, 2023, 13:17 +0800, guo jiwei , wrote:
> This is the second release candidate for Apache Pulsar, version 2.11.1.
>
> This release contains 214 commits by 53 contributors.
> https://github.com/apache/pulsar/compare/v2.11.0...v2.11.1-candidate-2
>
> CI for this release candidate
> https://github.com/Technoboy-/pulsar/pull/31
>
> *** Please download, test and vote on this release. This vote will stay open
> for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.1-candidate-2
>
> SHA-512 checksums:
>
> 98eddbc150ae4b832e0f37d7793ac62b3c1ac7fb98f7460292562d91c1dd580484e23786e02efdef6fb3f50ddcf5542e94ef1761dc0d943308bbba22b221c0dc
>
>
> ./apache-pulsar-2.11.1-bin.tar.gz
>
>
> 3dfe21d27c6668515ead3c55ef923a190df657ecabdbaa3451a060bd3670d6a709302e538c75b4b4f2e1d7b5289ee09ac3adba8b57ab59e98d13b4c0b57f8b83
>
>
> ./apache-pulsar-2.11.1-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1224/
>
> The tag to be voted upon:
> v2.11.1-candidate-2 (500833ac01fddf660e5ff1afccce1c7271fccd4c)
> https://github.com/apache/pulsar/releases/tag/v2.11.1-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> https://hub.docker.com/layers/mattison/pulsar-all/2.11.1-rc2/images/sha256-0a039a339f4ef5f44ed2d71b7684038d8d8bbe86375bd1421c55d83a6ae3d7b4
>
> https://hub.docker.com/layers/mattison/pulsar/2.11.1-rc2/images/sha256-cb8b9d860217145fa1c3ff197cf4c14f2470127fe714c609470137d1fa2f5951
>
> Please download the source package, and follow the
> release-candidate-validation doc to build
> and run the Pulsar standalone service.
> https://pulsar.apache.org/contribute/validate-release-candidate
>
> Since the metadata store is changed from ZK to RocksDB, the
> verification of the `stateful functions` needs to set the parameter
> "export PULSAR_STANDALONE_USE_ZOOKEEPER=1"
>
>
> Regards
> Jiwei Guo (Tboy)

Re: [DISCUSS] PIP-250: Add proxyVersion to CommandConnect

[mattisonchao]

Sorry for the late response.

Why do we need to make the broker aware of the proxy when, by normal software 
design, we should avoid coupling the concepts in the proxy and the broker? The 
previous authentication was for historical reasons, but we should not continue 
to introduce this coupling.

The proxy should be a separate component. Instead of continuing to couple the 
relevant proxy concepts into the broker, everyone should be a client to the 
broker.

Best,
Mattison
On Feb 25, 2023, 01:12 +0800, Michael Marshall , wrote:
> Great suggestions, Enrico.
>
> > It would be interesting to reject connections on the Proxy if the
> > client tries to set that field.
>
> I support making the proxy reject invalid input. We could also have
> the client reject connections where the client connect command
> includes `original_principal`, `original_auth_data`, or
> `original_auth_method`, since those are also only meant to be sent by
> the proxy.
>
> > On the broker there is no way to distinguish a proxy from a client, that's 
> > fair.
>
> We can reject these connections when authentication and authorization
> are enabled. My draft PR includes such logic.
>
> Thanks,
> Michael
>
> On Fri, Feb 24, 2023 at 7:29 AM Enrico Olivelli  wrote:
> >
> > Makes sense.
> >
> > It would be interesting to reject connections on the Proxy if the
> > client tries to set that field.
> > On the broker there is no way to distinguish a proxy from a client, that's 
> > fair.
> > But on the proxy it is not expected to see a connection from another proxy.
> >
> > +1
> >
> > Enrico
> >
> > Il giorno ven 24 feb 2023 alle ore 10:00 Zike Yang  ha 
> > scritto:
> > > >
> > > > Hi, Michael
> > > >
> > > > Thanks for initiating this PIP.
> > > >
> > > > +1
> > > >
> > > > BR,
> > > > Zike Yang
> > > >
> > > >
> > > > Zike Yang
> > > >
> > > > On Fri, Feb 24, 2023 at 12:16 PM Michael Marshall 
> > > >  wrote:
> > > > > >
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > In talking with Zike Yang on
> > > > > > https://github.com/apache/pulsar/pull/19540, we identified that it
> > > > > > would be helpful for the proxy to forward its own version when
> > > > > > connecting to the broker. Here is a related PIP to improve the
> > > > > > connection information available to operators.
> > > > > >
> > > > > > Issue: https://github.com/apache/pulsar/issues/19623
> > > > > > Implementation: https://github.com/apache/pulsar/pull/19618
> > > > > >
> > > > > > I look forward to your feedback!
> > > > > >
> > > > > > Thanks,
> > > > > > Michael
> > > > > >
> > > > > > Text of PIP copied below:
> > > > > >
> > > > > > ### Motivation
> > > > > >
> > > > > > When clients connect through the proxy, it is valuable to know which
> > > > > > version of the proxy connected to the broker. That information isn't
> > > > > > currently logged or reported in any easily identifiable way. The 
> > > > > > only
> > > > > > way to get information about the connection is to infer which proxy
> > > > > > forwarded a connection based on matching up the IP address in the
> > > > > > logs.
> > > > > >
> > > > > > An additional change proposed in the implementation is to log this 
> > > > > > new
> > > > > > information along with the `clientVersion`, `clientProtocolVersion`,
> > > > > > and relevant authentication role information. This information will
> > > > > > improve debug-ability and could also serve as a form of audit 
> > > > > > logging.
> > > > > >
> > > > > > ### Goal
> > > > > >
> > > > > > Improve the value of the broker's logs and metrics about connections
> > > > > > to simplify debugging and to make it easier for Pulsar operators to
> > > > > > understand how clients are connecting to their clusters.
> > > > > >
> > > > > > ### API Changes
> > > > > >
> > > > > > Add the following:
> > > > > >
> > > > > > ```proto
> > > > > > message CommandConnect {
> > > > > > // Other fields omitted
> > > > > > optional string proxy_version = 11; // Version of the proxy.
> > > > > > Should only be forwarded by a proxy.
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > ### Implementation
> > > > > >
> > > > > > Initial implementation: https://github.com/apache/pulsar/pull/19618
> > > > > >
> > > > > > ### Alternatives
> > > > > >
> > > > > > The `CommandAuthResponse` has a `client_version` field. It's 
> > > > > > possible
> > > > > > that someone would want this `proxy_version` field on that message.
> > > > > > However, I think we should not continue down the path of duplicating
> > > > > > fields in the connection handshake protocol.
> > > > > >
> > > > > > ### Anything else?
> > > > > >
> > > > > > Future work could include exposing this `proxyVersion` and
> > > > > > `clientVersion` information via Prometheus metrics.

Re: [DISCUSS] Dropping the StreamingDispatcher

[mattisonchao]

Totally agree with it. +1

Best
Mattison
On Apr 6, 2023, 10:53 +0800, Devin Bost , wrote:
> +1 since it can be pulled back up in git history if someone decides to do
> something with it to improve it at a later time.
>
> I also agree that it's a pain to maintain, and I don't know anyone using
> it. I've gone through some of those code paths, and I was concerned about
> divergence anyway.
>
> - Devin
>
>
> On Wed, Apr 5, 2023, 5:40 PM Michael Marshall  wrote:
>
> > If the code isn't being used or maintained, I support removing it. The
> > code will be available in the git history in case someone decides to
> > resurrect it.
> >
> > Thanks,
> > Michael
> >
> > On Wed, Apr 5, 2023 at 7:14 AM Enrico Olivelli 
> > wrote:
> > > >
> > > > Yunze,
> > > >
> > > > Il Mar 4 Apr 2023, 09:57 Yunze Xu  ha
> > scritto:
> > > >
> > > > > > If the flaky tests were the only concern, I think we can just 
> > > > > > disable
> > > > > > these tests.
> > > >
> > > >
> > > > My concern is not about the the flaky tests but a out maintenance of 
> > > > dead
> > > > code.
> > > >
> > > >
> > > >
> > > > Whatever, this config in `ServiceConfiguration` has
> > > > > > existed for a long time, though when it was introduced, the PIP rule
> > > > > > was not clear so there is no PIP for it.
> > > > > >
> > > >
> > > > I don't think it would work well in production, given the amount of
> > > > flakyness in the tests and the fact that nobody ever asked questions
> > about
> > > > it.
> > > >
> > > >
> > > > This is why I propose to drop the code now in Pulsar 3.0
> > > >
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > > > > Thanks,
> > > > > > Yunze
> > > > > >
> > > > > > On Tue, Apr 4, 2023 at 3:09 PM Gavin gao 
> > wrote:
> > > > > > > >
> > > > > > > > +1, I totally agree with this idea.
> > > > > > > >
> > > > > > > > Enrico Olivelli  于2023年4月4日周二 14:47写道：
> > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > > It has been a long time that we have in the Pulsar code a 
> > > > > > > > > > new
> > > > > > > > > > experimental Dispatcher implementation named 
> > > > > > > > > > StreamingDispatcher.
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/pulsar/pull/9056
> > > > > > > > > >
> > > > > > > > > > There are many flaky tests about that feature and I believe 
> > > > > > > > > > that it
> > > > > > > > > > has never been used in Production by anyone, because it 
> > > > > > > > > > happened a
> > few
> > > > > > > > > > times that we did some changes in the regular Dispatcher and
> > > > > > > > > > introduced bugs on the StreamingDispacther (usually 
> > > > > > > > > > manifested as
> > > > > > > > > > flaky tests)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I propose to drop the StreamingDispatcher code for Pulsar 
> > > > > > > > > > 3.0.
> > > > > > > > > > I don't think we need a PIP for this, it is an experimental 
> > > > > > > > > > code
> > that
> > > > > > > > > > was never delivered as a production ready feature.
> > > > > > > > > >
> > > > > > > > > > If anyone is aware of users please chime in.
> > > > > > > > > >
> > > > > > > > > > If anyone wants to sponsor that feature and objects in 
> > > > > > > > > > removing
> > this
> > > > > > > > > > dead code (that we still have to maintain) please help us in
> > > > > > > > > > completing the feature.
> > > > > > > > > >
> > > > > > > > > > On paper it is a very appealing feature, and I am 
> > > > > > > > > > disappointed in
> > > > > > dropping
> > > > > > > > > > it.
> > > > > > > > > > On the other hand, this is dead code that we have to 
> > > > > > > > > > maintain with
> > zero
> > > > > > > > > > benefit
> > > > > > > > > >
> > > > > > > > > > Thoughts ?
> > > > > > > > > >
> > > > > > > > > > Enrico
> > > > > > > > > >
> > > > > >
> >

Re: [VOTE] Pulsar Release 2.10.4 Candidate 3

[mattisonchao]

+1 (Binding)

• Built from the source package (maven 3.8.6 OpenJDK 11.0)
• Ran binary package standalone with pub/sub
• Ran docker image(pulsar-all) standalone with pub/sub
• Ran License check



Best,
Mattison
On Mar 27, 2023, 18:58 +0800, 丛搏 , wrote:
> +1(binding)
>
> Please ignore the previous reply, I read it wrong, sorry.
>
> +1 (binding)
>
> os: mac 12.3.1, Intel
> java: OpenJDK 17.0.1
>
> - Checked the signature
> - Checked LICENSE
> - Start standalone
> - Publish and consume messages
> - Verified Function and State Function
> - Verified Cassandra connector
> - Build from the source package (maven 3.8.6, openJDK 11.0.12)
> - Run a simple transaction check
>
> Thanks,
> Bo
>
>
> 丛搏  于2023年3月27日周一 18:20写道：
>
> >
> > Hi, Xiangying:
> >
> > I found in
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.4-candidate-3/connectors/
> > has two pulsar-io-debezium-mssql-2.10.4.nar
> >
> > Thanks,
> > Bo
> >
> > guo jiwei  于2023年3月23日周四 18:27写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > - Checked the signature
> > > > - Checked LICENSE
> > > > - Publish and consume messages
> > > > - Verified Function and State Function
> > > > - Verified Cassandra connector
> > > > - Build from the source package
> > > >
> > > >
> > > > Regards
> > > > Jiwei Guo (Tboy)
> > > >
> > > > On Wed, Mar 22, 2023 at 3:08 PM Xiangying Meng  
> > > > wrote:
> > > > > >
> > > > > > This is the third release candidate for Apache Pulsar, version 
> > > > > > 2.10.4.
> > > > > >
> > > > > > This release contains 111 commits by 35 contributors.
> > > > > > https://github.com/apache/pulsar/compare/v2.10.3...v2.10.4-candidate-3
> > > > > >
> > > > > > *** Please download, test and vote on this release. This vote will 
> > > > > > stay open
> > > > > > for at least 72 hours ***
> > > > > >
> > > > > > Note that we are voting upon the source (tag), binaries are 
> > > > > > provided for
> > > > > > convenience.
> > > > > >
> > > > > > Source and binary files:
> > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.10.4-candidate-3/
> > > > > >
> > > > > > SHA-512 checksums:
> > > > > > 59f0326643cca9ef16b45b4b522ab5a1c1d8dc32ac19897704f8231f9bd4cef02af722848646332db461a807daacc9cb87993b81dcf1429b1f23e3872a32
> > > > > > apache-pulsar-2.10.4-bin.tar.gz
> > > > > > 5b2adbf0d371b79b1dbe141f152848049d19924151fa8827057038d81833accd70cf67429cb003aedb8d44ee705ed0609d49757e74fed377dce77b09d49062e3
> > > > > > apache-pulsar-2.10.4-src.tar.gz
> > > > > >
> > > > > > Maven staging repo:
> > > > > > https://repository.apache.org/content/repositories/orgapachepulsar-1221/
> > > > > >
> > > > > > The tag to be voted upon:
> > > > > > v2.10.4-candidate-3
> > > > > > (e4898ac8eb37f698f29aa21e40a3abdda5489d45)
> > > > > > https://github.com/apache/pulsar/releases/tag/v2.10.4-candidate-3
> > > > > >
> > > > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > > > https://downloads.apache.org/pulsar/KEYS
> > > > > >
> > > > > > Docker images:
> > > > > >
> > > > > > 
> > > > > > https://hub.docker.com/layers/xiangyingmeng/pulsar/2.10.4/images/sha256-05bfb482c5b5aa66ac818651d8997745ac7d536ca0cb56bff8199a6de459ac45?context=repo
> > > > > >
> > > > > > 
> > > > > > https://hub.docker.com/layers/xiangyingmeng/pulsar-all/2.10.4/images/sha256-d4f3de64a8ec4a9039ac500bbf4a0efae9a9f1d4e0a58e11cab020276dc5e6b3?context=repo
> > > > > >
> > > > > > Please download the source package, and follow the README to build
> > > > > > and run the Pulsar standalone service.

Re: [VOTE] Pulsar Release 2.9.5 Candidate 2

[mattisonchao]

+1 (Binding)

• Built from the source package (maven 3.8.6 OpenJDK 11.0)
• Ran binary package standalone with pub/sub
• Ran docker image(pulsar-all) standalone with pub/sub
• Ran License check


Best,
Mattison
On Mar 28, 2023, 22:23 +0800, guo jiwei , wrote:
> +1 (binding)
>
> - Checked the signature
> - Start standalone
> - Publish and consume messages
> - Verified Function and State Function
> - Verified Cassandra connector
> - Build from the source package
>
>
>
> Regards
> Jiwei Guo (Tboy)
>
> On Tue, Mar 28, 2023 at 2:34 PM 丛搏  wrote:
> >
> > +1 (binding)
> >
> > os: mac 12.3.1, Intel
> > java: OpenJDK 17.0.1
> >
> > - Checked the signature
> > - Checked LICENSE
> > - Start standalone
> > - Publish and consume messages
> > - Verified Function and State Function
> > - Verified Cassandra connector
> > - Build from the source package (maven 3.8.6, openJDK 17.0.1)
> > - Run a simple transaction check
> >
> > Thanks,
> > Bo
> >
> > Cong Zhao  于2023年3月27日周一 23:09写道：
> > > >
> > > > This is the third release candidate for Apache Pulsar, version 2.9.5.
> > > >
> > > > This release contains 103 commits by 30 contributors.
> > > > https://github.com/apache/pulsar/compare/v2.9.4...v2.9.5-candidate-2
> > > >
> > > > *** Please download, test, and vote on this release. This vote will stay
> > > > open
> > > > for at least 72 hours ***
> > > >
> > > > Note that we are voting upon the source (tag), binaries are provided for
> > > > convenience.
> > > >
> > > > Source and binary files:
> > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.9.5-candidate-2/
> > > >
> > > > SHA-512 checksums:
> > > > 5e1d0c1b38441cdcb36a2f4e59ab9755b39a5c4a0136e078e91ab9bc2169016f195268692cafd6f13a45248dba2e97959b41f3cfbc8659e3cbd0bade0c954998
> > > > apache-pulsar-2.9.5-bin.tar.gz
> > > >
> > > > 72c9f47005636c6e629dd5117b15fdc13bfd9c7efe107be77a9d55b7dfcdda2f941003eb120ea8beeffe44c41bc41c385a2e5a9cb6540d2fe83a6d04ea53a7389d
> > > > apache-pulsar-2.9.5-src.tar.gz
> > > >
> > > > 73d286af64e189cf91c0511d360d98371b7ade1eec67bc6acf6ff766784e9e40388d3da8ae99a206369feaf398b254fff36e2206077041c37b8055ee7edde86eea
> > > > apache-pulsar-offloaders-2.9.5-bin.tar.gz
> > > >
> > > > Maven staging repo:
> > > > https://repository.apache.org/content/repositories/orgapachepulsar-1222/
> > > >
> > > > The tag to be voted upon:
> > > > v2.9.5-candidate-2 (c75c811ee48f51cf74f399f5b364bc1527186b34)
> > > > https://github.com/apache/pulsar/releases/tag/v2.9.5-candidate-2
> > > >
> > > > Pulsar's KEYS file containing PGP keys you use to sign the release:
> > > > https://downloads.apache.org/pulsar/KEYS
> > > >
> > > > Docker images:
> > > >
> > > > 
> > > > https://hub.docker.com/layers/czcoder/pulsar/2.9.5/images/sha256-c6d3435d5699cb3697ee2ddc4f8a45e0ac5e35d8aefd557e280b7cf91366b981?context=explore
> > > >
> > > > 
> > > > https://hub.docker.com/layers/czcoder/pulsar-all/2.9.5/images/sha256-a09a8e177ca7856c29dc8b9828cd293c6d44b473add20d1877bb3137b94a20c5?context=explore
> > > >
> > > > 
> > > > https://hub.docker.com/layers/czcoder/pulsar-grafana/2.9.5/images/sha256-c43a489c65cf6c407d6c3be6fc7a001227805b1aaa9413115cf55ba11a1e329f?context=explore
> > > >
> > > > Please download the source package, and follow the README to build
> > > > and run the Pulsar standalone service.
> > > >
> > > > Thanks
> > > > Cong Zhao

Re: [VOTE] Pulsar Client Go Release 0.10.0 Candidate 1

[mattisonchao]

+1 (binding)

• Built the project
• Ran the simple producer & consumer with pulsar 2.9.4, 2.10.3, 2.11.0


Best,
Mattison
On Mar 30, 2023, 17:39 +0800, Takeshi Kimura , wrote:
> +1 (non-binding)
>
> - verified checksum and signature
> - ran producer and consumer examples
>
> Regards,
> Takeshi Kimura
>
> -元のメッセージ-
> 送信元: Zike Yang 
> Reply-To: "dev@pulsar.apache.org" 
> 日付: 2023年3月27日 月曜日 21:23
> 宛先: "dev@pulsar.apache.org" 
> 件名: [VOTE] Pulsar Client Go Release 0.10.0 Candidate 1
>
> Hi everyone,
> Please review and vote on the release candidate #1 for the version
> 0.10.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> This is the first release candidate for Apache Pulsar Go client, version 
> 0.10.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar-client-go/milestone/12?closed=1
>
> Pulsar Client Go's KEYS file contains PGP keys we used to sign this release:
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS=05%7C01%7Ctakeshki%40yahoo-corp.jp%7C7fa931e704744220829608db2ebe1fcb%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638155166332495746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=sriR%2Bg7eOnCvQ8S5iKk26fND24GXgZXmXDbxcuYMieI%3D=0
>
> Please download these packages and review this release candidate:
> - Review release notes: https://github.com/apache/pulsar-client-go/pull/997
> - Download the source package (verify shasum, and asc) and follow the
> README.md to build and run the pulsar-client-go.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Source file:
> https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-client-go-0.10.0-candidate-1%2F=05%7C01%7Ctakeshki%40yahoo-corp.jp%7C7fa931e704744220829608db2ebe1fcb%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638155166332495746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=%2F64IHSR%2BcwQZXxJ%2BMsEag9DmMG0lt3IzHBLCLCnM150%3D=0
>
> The tag to be voted upon:
> v0.10.0
> https://github.com/apache/pulsar-client-go/tree/v0.10.0-candidate-1
>
> SHA-512 checksums:
> 5f26d95061eb535595043ffc23424d9913f11d1b80ca815bdd20577453f6e08c2a5dd8b729d82494c4f5d0260905b231218c6a437995d893c4c25efdb0bb
> apache-pulsar-client-go-0.10.0-src.tar.gz
>
> Zike Yang
>

Re: [DISCUSS] Cherry-pick #15121 into branch-2.10 to solve the issue sasl authentication failure

[mattisonchao]

I agree with cherry-picking PR 15121 to branch-2.10 and keep compatibility.


Best,
Mattison

On Mar 28, 2023, 19:52 +0800, Yubiao Feng 
, wrote:
> Hi community
>
> ### Summary
> The Admin client (`pulsar-admin`) and Java Client (PulsarAdmin) will throw
> Unauthorized Ex in both scenarios:
> - If there have more than one broker in a cluster( see issue 1 below ).
> - If authentication is enabled for both Pulsar-Proxy and Pulsar-Broker( see
> issue 2 below),
>
> ```
> bin/pulsar-admin topics stats persistent://public/default/tp1
> 2023-03-28T07:30:58,453+ [main] INFO
> org.apache.pulsar.client.impl.auth.AuthenticationSasl - JAAS loginContext
> is: PulsarAdmin.
> 2023-03-28T07:30:58,583+ [main] INFO
> org.apache.pulsar.common.sasl.JAASCredentialsContainer - successfully
> logged in.
> 2023-03-28T07:30:58,587+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh thread started.
> 2023-03-28T07:30:58,612+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - Client principal is "
> pulsar-ad...@sn.io".
> 2023-03-28T07:30:58,613+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - Server principal is
> "krbtgt/sn...@sn.io".
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT valid starting at:
> Tue Mar 28 07:30:58 UTC 2023
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT expires:
> Wed Mar 29 07:30:58 UTC 2023
> 2023-03-28T07:30:58,617+ [pulsar-tgt-refresh-thread] INFO
> org.apache.pulsar.common.sasl.TGTRefreshThread - TGT refresh sleeping
> until: Wed Mar 29 03:12:29 UTC 2023
> 2023-03-28T07:30:59,861+ [main] INFO
> org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using
> JAAS/SASL/GSSAPI auth to connect to server Principal broker/pulsar03,
> HTTP 401 Unauthorized
> Reason: HTTP 401 Unauthorized
> ```
>
> And I want to cherry-pick https://github.com/apache/pulsar/pull/15121 into
> branch-2.10 to fix it.
>
> ### Background
> When using Kerberos for authentication, Pulsar works like this:
> - client: init ticket
> - request to broker
> - broker identifies the client (Broker can confirm the ticket is valid by
> Kerberos)
> - sends a token(we call it sasl_role_token) to the client ( at this moment,
> the session is successfully created )
> - then the client will be authenticated through sasl_role_token, do not use
> Kerberos anymore.
>
> The `sasl_role_token` is generated by this logic: `Sha512(saslRoleName,
> ${secret})`, we call the `secret` sasl_sign_secret.
> In version `2.10.x`, the variable `secret` is a random string initialized
> when the broker starts.
>
> ### Issue 1
> If a cluster includes two brokers, and a topic `public/default/tp1` is
> owned by broker-0. We will get an error when we call `pulsar-admin topics
> stats public/default/tp1` to broker-1.
>
> The whole process goes like this:
> - client succeeds in authentication and gets a token from broker-1
> - broker-1 tells the client to redirect to broker-0
> - client request to broker-0 carries the sasl_role_token generated by
> broker-1
> - broker-0 can not decode the sasl_role_token, because it has differ secret
> of broker-1, and responses 401
>
> ### Issue 2
> After authentication is enabled for both Pulsar-Proxy and Pulsar-Broker,
> the error occurs as follows
> - client succeeds in authentication and gets a token from Pulsar Proxy
> - proxy forwards the request to broker
> - the broker can not decode the `sasl_role_token`, because it has differed
> secret of Pulsar Proxy, and responses 401
>
> ### solutions
> There have two solutions to solve this issue:
>
> Solution 1
> - The client saves different tokens for different servers(e.g. ["broker-0",
> "broker-1", "pulsar-proxy"]) so servers will receive the tokens issued by
> each other, then we can fix Issue 1.
> - Proxy and Broker do not enable authentication simultaneously, then we can
> fix Issue 2.
>
> Solution 2
> - Make `sasl_sign_secret` configurable. Users can configure this variable
> to the same value, then multi servers can decode every
> `sasl_role_token.` PR #15121 does this.
>
> I'd prefer Solution 2 because it is already in the master branch, so I want
> to cherry-pick #15121 into branch-2.10.
>
> ### Forward Compatibility
> In PR #15121, the config `sasl_sign_secret` is a new item in config files.
> Since it is required, users will get a system error if does not set it. To
> ensure forward compatibility, we can make this variable optional in
> branch-2.10
>
>
> Thanks
> Yubiao Feng

Re: Release 2.11.1

[mattisonchao]

+1,
Thanks for your work!

Best,
Mattison
On Mar 15, 2023, 15:24 +0800, PengHui Li , wrote:
> +1
>
> Just a reminder, this one [0] might be a blocker for the subsequent patch
> releases since it has introduced a new rule of the rule_name.
> Here [1] is the discussion under the private mailing thread.
> IMO, we should revert this change first before starting the release.
>
> And I also hope more PMCs can follow this topic [1], providing your
> thoughts.
>
> Thanks,
> Penghui
>
> [0]
> https://github.com/apache/pulsar/commit/c7eabc9391361508fc222469479bfea83d5d0c64
> [1] https://lists.apache.org/thread/q1k1krfjr8njdf8t3sy1ht4vg67s4x71
>
> On Wed, Mar 15, 2023 at 12:25 PM Haiting Jiang 
> wrote:
>
> > +1
> >
> > Thanks,
> > Haiting
> >
> > On Wed, Mar 15, 2023 at 11:46 AM Yubiao Feng
> >  wrote:
> > > >
> > > > +1
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> > > > On Wed, Mar 15, 2023 at 11:43 AM guo jiwei  wrote:
> > > >
> > > > > > Hello, Pulsar community:
> > > > > >
> > > > > > I'd like to propose releasing Apache Pulsar 2.11.1. It's been about
> > > > > > two months since 2.11.0 was released.
> > > > > >
> > > > > > There are about over 200 PRS [0] cherry-picked to branch-2.11 since
> > > > > > the last release.
> > > > > >
> > > > > > If you have any important fixes or any questions, please reply to 
> > > > > > this
> > > > > > email and we will evaluate whether to include them in 2.11.1
> > > > > >
> > > > > >
> > > > > > [0]
> > > > > >
> > https://github.com/apache/pulsar/pulls?page=9=is%3Apr+label%3Arelease%2F2.11.1+-label%3Acherry-picked%2Fbranch-2.11.1+is%3Amerged
> > > > > >
> > > > > >
> > > > > >
> > > > > > Regards
> > > > > > Jiwei Guo (Tboy)
> > > > > >
> >

[DISCUSS] Critical problem report - session notification thread deadlock

[mattisonchao]

Hi, All

We found a critical problem that will cause the pulsar cluster to part “deaf” 
status. the broker can’t receive zookeeper session notification to revalidate 
namespace bundle ownership and leader election. That means it will cause one 
topic may have two owner brokers, and the leader election problem. etc

Blast radius:
Since this problem was introduced by 
https://github.com/apache/pulsar/pull/17401. The releases as follows may affect:

2.8.5
2.9.4
2.10.3
2.11.0

Workaround:

Restart broker

The fix is over here: https://github.com/apache/pulsar/pull/19754

We can avoid upgrading to the above version and wait for the latest progress. I 
will continue to push for a solution to this problem.

Please correct me if I got something wrong. thanks!


Best,
Mattison

Re: Please stop cherry-picking (breaking) changes to the released branches

[mattisonchao]

+1000,

We should pay attention on two places:

1. Why add release/xxx label to the breaking change PR ?
2. We should recheck the context when committer cherry-pick PRs.


Best,
Mattison
On Feb 27, 2023, 23:27 +0800, Enrico Olivelli , wrote:
> Hello Committers,
> I believe that we should stop cherry-picking breaking changes like [1]
> to released branches.
> Really, this is something that we cannot do.
>
> When you decide to cherry-pick a commit to a "stable branch",
> currently branch-2.8, branch-2.9, branch-2.10 and branch-2.11 you
> always have to think about these things:
> - is it a breaking change ?
> - is it really needed ?
> - could it mine the stability of the branch ?
>
> The answer is usually that you can cherry-pick a change only if it
> falls into these categories:
> - there is a security hole to fix (in this case the PMC has to deal
> with it, and usually this is done not publicly)
> - there is a bad bug that cause data loss or other serious problems
>
> I have sent this message a few other times in the past.
> We, the Pulsar community, are responsible for the stability of the
> project and product that our users use in production.
>
> Even if you think that something that could "improve the performance"
> or "do something better" is appealing you always have to keep in mind
> that the risk of breaking something that is stable is too high in
> respect to the gain in terms of performances or anything else.
>
> Improvements should go only to the master branch, and users will
> benefit from them when we will cut a release.
>
> This is a free OSS project on which many users count on.
>
> If you are eager to see a performance improvement in your system, then
> this is fine,
> this is OSS and you can legally have a fork and cherry-pick the
> patches and build it on your own.
> This is the reason why OSS is cool.
> But if you are able to cherry-pick a patch you are also able to
> maintain your fork and fix any problems if the patch caused a
> regression.
>
> Most of the consumers of OSS products rely on us because they don't
> have enough engineering resources to maintain such a project by
> themselves.
>
> They trust us and they won't scan a list of tens of commits in order
> to double check if the upgrade will change the behaviour of their
> applications.
>
> This is Pulsar momentum, let's do our best to fulfill the expectations
> of the companies that are adopting our project.
>
> Enrico
>
> [1] https://github.com/apache/pulsar/pull/19640#pullrequestreview-1315805022

Re: [DISCUSS] Change PIP template

[mattisonchao]

+1

Best,
Mattison
On Feb 26, 2023, 23:02 +0800, Dave Fisher , wrote:
> Excellent proposal.
>
> Inline below.
>
> Sent from my iPhone
>
> > On Feb 26, 2023, at 3:19 AM, Asaf Mesika  wrote:
> >
> > Hi,
> >
> > I would like to suggest two changes I'd like to make to the PIP design
> > template:
> > 1. Remove the form - just have a markdown template fill the issue body as
> > it is created.
> > 2. Change the PIP template structure
> >
> > == Removing the form
> >
> > Today, when you want to submit a PIP, you are required to fill out a form
> > with boxes composed of 3-4 lines length.
> > It's not good because:
> > * It broadcasts to the author: we want a very small PIP, something that
> > fits those small boxes.
> > * It makes the PIP look like a bug, where you fill out fields.
> > * It doesn't allow having H2 headings, only H1 headings, thus limiting the
> > structure.
> >
> > A PIP is a design essentially, something 1-3 pages long. Thus,
> > people take the time to write it down. Preferably, they copy paste the body
> > of the PIP issue, and use it to fill in sections.
> >
> > My suggestion is to define an issue template using only markdown, without a
> > form.
> >
> > == Changing PIP Structure
> >
> > Today the structure of the PIP doc (pasted below), is missing a section and
> > generally aims to jump directly into API changes / code / implementation.
> > This results in lots of back and forth emails in an attempt to get the
> > following essentials:
> > * All required background knowledge to understand the proposal
> > * A high level overview of the proposed solution
> > * Understanding how this proposal will be monitored
> > * What steps exactly I need to take if I revert to the previous version.
> >
> > The structure I propose below aims to reduce that friction and get all PIP
> > aligned to provide that information.
> >
> > === Today's structure
> >
> > # Motivation
> > * "Explain why this change is needed, what benefits it would bring to
> > Apache Pulsar and what problem it's trying to solve."
> > # Goal
> > * "Define the scope of this proposal. Given the motivation stated above,
> > what are the problems that this proposal is addressing and what other items
> > will be considering out of scope, perhaps to be left to a different PIP."
> > # API Changes
> > * "Illustrate all the proposed changes to the API or wire protocol, with
> > examples of all the newly added classes/methods, including Javadoc"
>
> Yes this is important as api and similar changes are what triggers requiring 
> small pips.
>
> > # Implementation
> > * "This should be a detailed description of all the changes that are
> > expected to be made. It should be detailed enough that any developer that
> > is familiar with Pulsar internals would be able to understand all the parts
> > of the code changes for this proposal."
> > * "This should also serve as documentation for any person that is trying to
> > understand or debug the behavior of a certain feature."
> > # Alernatives
> > * "If there are alternatives that were already considered by the authors
> > or, after the discussion, by the community, and were rejected, please list
> > them here along with the reason why they were rejected"
> > # Anything else?
> >
> >
> > === My suggestion
> >
> > # Motivation and Background information
> > * Give a high level explanation on all concepts you will be using
> > throughout this document. For example, if you want to talk about Persistent
> > Subscriptions, explain briefly (1 paragraph) what this is. If you're going
> > to talk about Transaction Buffer, explain briefly what this is. If you're
> > going to change something specific, that goes into a bit more detail about
> > it and how it works. The Litmus test: I can read the design document and
> > understand the problem statement and what you plan to change *without*
> > resorting to a couple of hours of code reading just to start having a high
> > level understanding of the change.
> > * Provide links where possible if a person wants to dig deeper into the
> > background information.
> > * Explain what is the problem you're trying to solve - current situation.
> > * This section is the "Why" of your proposal.
> >
> > # Goals
> > ## Scope
> > * Describe the goals of your proposal, and why it benefits Apache Pulsar
> > ## Out of Scope
> > * Describe what you have decided to keep out of scope, perhaps left for a
> > different PIP/s.
> >
> > # High-level Design
> > * Describe in high level, end-to-end, the solution. This should be a few
> > paragraphs long as a guideline.
> > * Reading this would allow me to understand the solution from a bird's eye
> > view, end to end.
> > * DON'T put all the design in a Google Doc and share the link here, as it
> > won't last the test of time.
>
> +1000! A Google doc hides details and obfuscates the design and motivation. 
> Google docs are not searchable within GitHub.
>
> We should reject all new PIPs that include Google Docs.
>
> Best,
> Dave
>
> >
> > 

Re: [DISCUSS] Release 2.9.5

[mattisonchao]

Hi, All

I am sorry, Cong Zhao will takeover this release.

Best,
Mattison
On Feb 21, 2023, 15:11 +0800, Haiting Jiang , wrote:
> +1
>
> Haiting
>
> On Tue, Feb 21, 2023 at 12:11 PM Hang Chen  wrote:
> >
> > +1
> >
> > Thanks,
> > Hang
> >
> > Enrico Olivelli  于2023年2月20日周一 21:11写道：
> > > >
> > > > +1
> > > >
> > > > Enrico
> > > >
> > > > Il giorno lun 20 feb 2023 alle ore 10:41 Zike Yang  ha 
> > > > scritto:
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > Thanks,
> > > > > > Zike Yang
> > > > > >
> > > > > > On Mon, Feb 20, 2023 at 12:57 PM  wrote:
> > > > > > > >
> > > > > > > > Hello, Pulsar community:
> > > > > > > >
> > > > > > > > I'd like to propose releasing Apache Pulsar 2.9.5. It's been 
> > > > > > > > about
> > > > > > > > two months since 2.9.4 was released.
> > > > > > > >
> > > > > > > > There are 54 PRs [0] needed to cherry-pick in branch-2.9. I will
> > > > > > > > cherry-pick these PRs for branch-2.9. Exclude some PRs that 
> > > > > > > > merge
> > > > > > > > directly into branch-2.9.
> > > > > > > >
> > > > > > > > There are 21 PRs [1] opened. I'll follow up on each of those 
> > > > > > > > PRs to
> > > > > > > > see if they will be completed soon or will need to be pushed to 
> > > > > > > > 2.9.6
> > > > > > > >
> > > > > > > > If you have any important fixes or any questions, please reply 
> > > > > > > > to this
> > > > > > > > email, and we will evaluate whether to include them in 2.9.5
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Mattison
> > > > > > > >
> > > > > > > > [0] 
> > > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.5+-label%3Acherry-picked%2Fbranch-2.9+is%3Aclosed+
> > > > > > > > [1] 
> > > > > > > > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.9.5

Re: [VOTE][PIP-242] Topic name restriction

[mattisonchao]

This PIP has passed voting. 5 (binding) and 4 (non-binding)

+1 (binding)

• Yunze Xu
• Penghui Li
• Guo jiwei
• Haiting Jiang
• Cong Bo


+1 (non-binding)

• Asaf Mesika
• Cong Zhao
• Zike Yang
• avinash kala



Thanks to you all, Let's meet at the next PIP.

Best,
Mattison
On Feb 18, 2023, 16:58 +0800, mattisonc...@gmail.com, wrote:
> Hi, All
>
> After a fascinating discussion, I would start the vote of PIP-242.
>
> We have chosen to drop out the `system topic` related improvement to another 
> PIP. Therefore, the current version is simple enough and it has a clear 
> boundary.
>
> Please leave +1/-1 in this thread to join the vote. and feel free to leave 
> any concerns.
>
> Thanks to you guys.
>
> Best,
> Mattison
>
> References:
>
> • PIP https://github.com/apache/pulsar/issues/19239
> • Discussion https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m
>
>

[DISCUSS] Release 2.9.5

[mattisonchao]

Hello, Pulsar community:

I'd like to propose releasing Apache Pulsar 2.9.5. It's been about
two months since 2.9.4 was released.

There are 54 PRs [0] needed to cherry-pick in branch-2.9. I will
cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
directly into branch-2.9.

There are 21 PRs [1] opened. I'll follow up on each of those PRs to
see if they will be completed soon or will need to be pushed to 2.9.6

If you have any important fixes or any questions, please reply to this
email, and we will evaluate whether to include them in 2.9.5


Best,
Mattison

[0] 
https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.5+-label%3Acherry-picked%2Fbranch-2.9+is%3Aclosed+
[1] 
https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.9.5

[VOTE][PIP-242] Topic name restriction

[mattisonchao]

Hi, All

After a fascinating discussion, I would start the vote of PIP-242.

We have chosen to drop out the `system topic` related improvement to another 
PIP. Therefore, the current version is simple enough and it has a clear 
boundary.

Please leave +1/-1 in this thread to join the vote. and feel free to leave any 
concerns.

Thanks to you guys.

Best,
Mattison

References:

• PIP https://github.com/apache/pulsar/issues/19239
• Discussion https://lists.apache.org/thread/oz79m0f2nw059jctq4cmms74yq5n2l1m

Re: [PROPOSAL] Roadmap for 3.0 release

[mattisonchao]

+1
On Feb 18, 2023, 14:56 +0800, Michael Marshall , wrote:
> +1 - this timeline sounds even better :)
>
> On Sat, Feb 18, 2023 at 12:41 AM Matteo Merli  wrote:
> >
> > Ups,
> >
> > I started from the release date I was meaning April for the RCs:
> >
> > * Tue - 2023-04-11
> > * Tue - 2023-04-18 - RC-2
> > * Tue - 2023-04-25 - RC-3
> > * Tue - 2023-05-02 - Announce 3.0 Release
> >
> > Sorry for the mixup!
> >
> > --
> > Matteo Merli
> > 
> >
> >
> > On Fri, Feb 17, 2023 at 10:39 PM Dave Fisher  wrote:
> >
> > > > +1.
> > > >
> > > > I think that there is a typo. See below.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > > On Feb 17, 2023, at 2:44 PM, Matteo Merli  wrote:
> > > > > >
> > > > > > Since the LTS release model has been formally approved, I'm 
> > > > > > proposing
> > > > > > the following schedule for the release:
> > > > > >
> > > > > > * Tue - 2023-05-11
> > > > > > - RC-1
> > > > > > - Code Freeze -- Only critical fixes will be merged in the 3.0
> > > > > > release branch. Contributors should plan to have all the changes 
> > > > > > merged
> > > > in
> > > > > > before this date. Exceptions should be extremely rare and strongly
> > > > > > motivated.
> > > > > >
> > > > > > * Tue - 2023-05-18 - RC-2
> > > > > > * Tue - 2023-05-25 - RC-3
> > > > > > * Tue - 2023-05-02 - Announce 3.0 Release
> > > >
> > > > You meant June 2, 2023?
> > > >
> > > > Best,
> > > > Dave
> > > > > >
> > > > > > These dates will be published on the website to present users with a
> > > > > > "roadmap" and we should commit to and respect these dates.
> > > > > >
> > > > > > I also wanted to propose trying out a model where we have 3 release
> > > > > > managers for all major releases.
> > > > > >
> > > > > > The reasoning behind this is for this small group of people to
> > > > collaborate
> > > > > > and divide the tasks for the release: merging patches from the 
> > > > > > "master"
> > > > > > branch, preparing RC, and testing.
> > > > > >
> > > > > > Since everyone also has other work duties and unexpected tasks that 
> > > > > > can
> > > > pop
> > > > > > up at any time, it will help to have redundancy in the 
> > > > > > release-management
> > > > > > "team", so that we can release on the exact dates.
> > > > > >
> > > > > > Thanks,
> > > > > > Matteo
> > > > > >
> > > > > > --
> > > > > > Matteo Merli
> > > > > > 
> > > >
> > > >

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

Hi, All

After discussing with Enrico and Michael offline.
I will split the discussed topic into two PIP.

1. Topic name restrictions
a. `-partition-` keyword.
b. enable topic name character pattern.
2. System topic
a. System topic name pattern.
b. System topic authorisation.
c. ...

In this approach, we will get a clear boundary and avoid going off the initial 
scope.

Since we don't have any question about the first scope. I will start vote next 
week.

Thanks to all participant.

Best,
Mattison



On Feb 18, 2023, 14:24 +0800, Michael Marshall , wrote:
> I support breaking this into two PIPs. It was my fault the two PIPs
> were merged in the first place. I am sorry if I created any confusion.
> My intention was only to point out that names are a meaningful way to
> simplify logic, and we should reserve certain names for Pulsar's own
> usage with a well defined pattern so that we can simplify lifecycle
> operations.
>
> Thanks,
> Michael
>
> On Fri, Feb 17, 2023 at 1:55 AM Enrico Olivelli  wrote:
> >
> > Mattison,
> >
> > Il giorno gio 16 feb 2023 alle ore 00:27  ha 
> > scritto:
> > > >
> > > > > > I am sorry but I am not sure that this is enough to 
> > > > > > preventreads/writes from unallowed clients.
> > > > IMO, We can consider the authorisation part in another PIP because We 
> > > > are just focusing on adding the topic name constraint of topic creation.
> > > >
> > > > Maybe we can use another PIP to clearify all of system topic's 
> > > > behaviour, like authorisation something.
> > > > e.g. we just allow superusers to read/write the data to that system 
> > > > topic.
> > > > > > We should elaborate more on this topic on the PIP
> > > > I will add the internal system topic creation logic in the PIP.
> > Why do you think that this is enough ?
> >
> > I think that we are going off the initial scope of the PIP.
> > The initial problem is about preventing clients from creating topics
> > that contain the "-partition-" keyword.
> >
> > I totally agree that there must be a clear way to distinguish topics
> > that are not meant to be accessed by "regular clients".
> >
> > The answer is in Micheal's words: only super users are allowed to
> > access topics that are not meant to be accessed by clients.
> > Broker to Broker communications are always running with a "super user"
> > role, so it is not a problem.
> >
> > BTW I wonder if it is better to narrow down the scope of the PIP and
> > go back to "-partition-"
> >
> >
> > Enrico
> >
> >
> > > >
> > > > Best,
> > > > Mattison
> > > > On Feb 16, 2023, 00:41 +0800, Enrico Olivelli , 
> > > > wrote:
> > > > > > Il giorno mer 15 feb 2023 alle ore 17:07  
> > > > > > ha scritto:
> > > > > > > >
> > > > > > > > Hi Enrico
> > > > > > > >
> > > > > > > > I think it's a good question. We can introduce a new method in 
> > > > > > > > the BrokerService to help brokers create the topic internally 
> > > > > > > > first(maybe just metadata is enough), and then to use a pulsar 
> > > > > > > > client to connect to it.
> > > > > >
> > > > > > I am sorry but I am not sure that this is enough to prevent
> > > > > > reads/writes from unallowed clients.
> > > > > > We should elaborate more on this topic on the PIP
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Mattison
> > > > > > > > On Feb 16, 2023, 00:01 +0800, Enrico Olivelli 
> > > > > > > > , wrote:
> > > > > > > > > > > > I have one question (apologies for the top posting).
> > > > > > > > > > > >
> > > > > > > > > > > > The Broker (and the other Pulsar components) use the 
> > > > > > > > > > > > regular Pulsar
> > > > > > > > > > > > client to connect to "system topics"
> > > > > > > > > > > > and in general they use the Pulsar wire protocol.
> > > > > > > > > > > >
> > > > > > > > > > > > The question is "how do you distinguish an internal 
> > > > > > > > > > > > component from a
> > > > > > > > > > > > user component ?"
> > > > > > > > > > > > How can you say that the broker is able to connect to a 
> > > > > > > > > > > > system topic
> > > > > > > > > > > > and any other client cannot do it ?
> > > > > > > > > > > >
> > > > > > > > > > > > Enrico
> > > > > > > > > > > >
> > > > > > > > > > > > Il giorno mer 15 feb 2023 alle ore 15:38 
> > > > > > > > > > > >  ha scritto:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Asaf
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There is a link to introduce the dynamic 
> > > > > > > > > > > > > > > > configuration.
> > > > > > > > > > > > > > > > https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Mattison
> > > > > > > > > > > > > > > > On Feb 14, 2023, 17:06 +0800, Asaf Mesika 
> > > > > > > > > > > > > > > > , wrote:
> > > > > > > > > > > > > > > > > > > > > > > > On 

Re: [DISCUSS] Switch the client complier to JDK 11 or 17 from JDK 8

[mattisonchao]

-1

The reason is the same as Yunze mentioned. It's a kind of break for the user. 
We can't force our user upgrade the application JDK version by pulsar's bug. 
That will make our users frustrated.

Plus, the pulsar LTS is 3.0. I think it's better to keep the JDK 1.8 in this 
version. as users will most likely use this LTS version for long time.

Best,
Mattison
On Feb 16, 2023, 14:49 +0800, Yunze Xu , wrote:
> -1
>
> Upgrading the client is harder than upgrading the broker. Pulsar
> usually acts as a service to serve many lines of business and only a
> single team is responsible to maintain the server side. It would be
> hard to push all the client sides to upgrade the JDK.
>
> Instead of upgrading the minimum required JDK of the client, it's
> better to open a separate project if you think the client with JDK 17
> could bring significant performance and security improvements.
>
> Thanks,
> Yunze
>
> On Wed, Feb 15, 2023 at 1:08 AM Zixuan Liu  wrote:
> >
> > Hi all,
> >
> > We are using JDK 17 to compile all the components. For the main component,
> > the broker and proxy require JDK 17, and the client requires JDK 8.
> >
> > In Pulsar 3.x, we should keep up with modern JDKS for all components, and
> > over time, I believe many users have adopted JDK 11/17, which provides
> > significant performance and security improvements, see
> > https://learn.microsoft.com/en-us/java/openjdk/reasons-to-move-to-java-11
> >
> > Switching to JDK17 is a bit radical, and we still need to consider this.
> >
> > Thanks,
> > Zixuan

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[mattisonchao]

+1

Since web pages are more easily and public.

Best
Mattison
On Feb 16, 2023, 07:58 +0800, Christophe Bornet , wrote:
> +100
> Also note that for good or bad reasons, the number of questions on
> StaOverflow is often used as a metric for the popularity of a project.
>
> Le mer. 15 févr. 2023 à 13:12, Kiryl Valkovich 
> a écrit :
>
> > Hi everyone! Enrico Olivelli asked me to repost it to the mailing list.
> >
> > --- Me
> > I’m very worried that good answers from David Kjerrumgaard and others
> > won’t be googleable because it’s in Slack. To make any search you need to
> > pay a monthly subscription.
> >
> > In my opinion, it would be wiser to make StackOverflow, Discuss, or GitHub
> > discussions a place for questions. And strictly redirect people who ask any
> > question in Slack to the right place.
> > In GitHub discussions, you also can manage categories, as well as you can
> > manage channels in Slack.
> >
> > Subscription to a specific category is coming soon.
> > https://github.com/community/community/discussions/3951#discussioncomment-3879939
> > The most important that it’s searchable.
> >
> > Examples:
> > https://github.com/community/community/discussions
> > https://github.com/apache/superset/discussions
> >
> > --- Later me to David Kjerrumgaard
> > What do you think about the following workflow:
> > 1. The user asks a new question in Slack.
> > 2. You click the three dots button on his message -> “Convert to a GitHub
> > discussion”.
> > 3. The bot creates a new GitHub discussion in the “Q” category with the
> > original message’s content.
> > 4. The bot converts the original message to a thread and posts a message
> > like:
> > > > In order to keep history and make your question searchable by other
> > Apache Pulsar users, it has been converted to a GitHub discussion.
> > > > Further conversation is available by this link:
> > https://github.com/apache/pulsar/discussions/18766.
> > 5. You answer the question on GitHub.
> >
> > How it may work:
> > https://api.slack.com/best-practices/blueprints/actions-and-dialogs
> > I can try to implement it makes any sense for you. (edited)
> >
> > Full original thread:
> > https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1676449603034309
> >
> >

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

> I am sorry but I am not sure that this is enough to preventreads/writes from 
> unallowed clients.
IMO, We can consider the authorisation part in another PIP because We are just 
focusing on adding the topic name constraint of topic creation.

Maybe we can use another PIP to clearify all of system topic's behaviour, like 
authorisation something.
e.g. we just allow superusers to read/write the data to that system topic.
> We should elaborate more on this topic on the PIP
I will add the internal system topic creation logic in the PIP.

Best,
Mattison
On Feb 16, 2023, 00:41 +0800, Enrico Olivelli , wrote:
> Il giorno mer 15 feb 2023 alle ore 17:07  ha scritto:
> >
> > Hi Enrico
> >
> > I think it's a good question. We can introduce a new method in the 
> > BrokerService to help brokers create the topic internally first(maybe just 
> > metadata is enough), and then to use a pulsar client to connect to it.
>
> I am sorry but I am not sure that this is enough to prevent
> reads/writes from unallowed clients.
> We should elaborate more on this topic on the PIP
>
> Enrico
>
> >
> > WDYT?
> >
> >
> > Best,
> > Mattison
> > On Feb 16, 2023, 00:01 +0800, Enrico Olivelli , wrote:
> > > > I have one question (apologies for the top posting).
> > > >
> > > > The Broker (and the other Pulsar components) use the regular Pulsar
> > > > client to connect to "system topics"
> > > > and in general they use the Pulsar wire protocol.
> > > >
> > > > The question is "how do you distinguish an internal component from a
> > > > user component ?"
> > > > How can you say that the broker is able to connect to a system topic
> > > > and any other client cannot do it ?
> > > >
> > > > Enrico
> > > >
> > > > Il giorno mer 15 feb 2023 alle ore 15:38  ha 
> > > > scritto:
> > > > > >
> > > > > > Hi Asaf
> > > > > >
> > > > > > There is a link to introduce the dynamic configuration.
> > > > > > https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration
> > > > > >
> > > > > > Best,
> > > > > > Mattison
> > > > > > On Feb 14, 2023, 17:06 +0800, Asaf Mesika , 
> > > > > > wrote:
> > > > > > > > > > On Tue, Feb 14, 2023 at 3:46 AM  
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > > > Hi, Asaf
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Welcome to join this discussion.
> > > > > > > > > > > > > > > > > > > > > > You mean that allows the *system* 
> > > > > > > > > > > > > > > > > > > > > > to use it when it's a partitioned
> > > > > > > > > > > > > > topic?
> > > > > > > > > > > > > > Sorry, I didn't get your point. What do you mean by 
> > > > > > > > > > > > > > *system*?
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This sentence was a reply to:
> > > > > > > > > >
> > > > > > > > > > 2. Make the `-partition-` string the keyword. That allows 
> > > > > > > > > > the user to use
> > > > > > > > > > > > > > it when it's a partitioned topic.
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I wanted to say that this sentence should be:
> > > > > > > > > > Make the `-partition-` string the keyword, that allows the 
> > > > > > > > > > *system* to use
> > > > > > > > > > it when it's a partitioned topic.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Why postfix of `__`?Why uppercase 
> > > > > > > > > > > > > > > > > > > > > > ?Maybe `__system__`?
> > > > > > > > > > > > > > Yes, That is a key point that I want to discuss in 
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > thread. `__system__` is good for me.
> > > > > > > > > > > > > > > > > > > > > > Can you please elaborate what it 
> > > > > > > > > > > > > > > > > > > > > > means to make it dynamic exactly?
> > > > > > > > > > > > > > Sorry, I will refine it. it means we can update 
> > > > > > > > > > > > > > this configuration
> > > > > > > > > > > > > > dynamically. (using rest api or sth)
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I'm unfamiliar with how Pulsar supports dynamic 
> > > > > > > > > > configuration. I would
> > > > > > > > > > love it if you can share a link or explain it briefly, thus 
> > > > > > > > > > explaining what
> > > > > > > > > > exactly you are going to change to support dynamic 
> > > > > > > > > > configuration.
> > > > > > > > > >
> > > > > > > > > > > > > > General question: In the last thread you said 
> > > > > > > > > > > > > > something about
> > > > > > > > > > > > > > configurablerules, etc? You decided not to use this 
> > > > > > > > > > > > > > idea?
> > > > > > > > > > > > > > IMO, That idea is an advanced feature. we may need 
> > > > > > > > > > > > > > more time to discuss
> > > > > > > > > > > > > > the details and for the topic name restriction, 
> > > > > > > > > > > > > > maybe we don't have strong
> > > > > > > > > > > > > > reason to use that.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We can introduce this advanced feature 

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

Hi Enrico

I think it's a good question. We can introduce a new method in the 
BrokerService to help brokers create the topic internally first(maybe just 
metadata is enough), and then to use pulsar client connect to it.

WDYT?


Best,
Mattison
On Feb 16, 2023, 00:01 +0800, Enrico Olivelli , wrote:
> I have one question (apologies for the top posting).
>
> The Broker (and the other Pulsar components) use the regular Pulsar
> client to connect to "system topics"
> and in general they use the Pulsar wire protocol.
>
> The question is "how do you distinguish an internal component from a
> user component ?"
> How can you say that the broker is able to connect to a system topic
> and any other client cannot do it ?
>
> Enrico
>
> Il giorno mer 15 feb 2023 alle ore 15:38  ha scritto:
> >
> > Hi Asaf
> >
> > There is a link to introduce the dynamic configuration.
> > https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration
> >
> > Best,
> > Mattison
> > On Feb 14, 2023, 17:06 +0800, Asaf Mesika , wrote:
> > > > On Tue, Feb 14, 2023 at 3:46 AM  wrote:
> > > >
> > > > > > Hi, Asaf
> > > > > >
> > > > > > Welcome to join this discussion.
> > > > > > > > > > You mean that allows the *system* to use it when it's a 
> > > > > > > > > > partitioned
> > > > > > topic?
> > > > > > Sorry, I didn't get your point. What do you mean by *system*?
> > > > > >
> > > >
> > > > This sentence was a reply to:
> > > >
> > > > 2. Make the `-partition-` string the keyword. That allows the user to 
> > > > use
> > > > > > it when it's a partitioned topic.
> > > > > >
> > > >
> > > > I wanted to say that this sentence should be:
> > > > Make the `-partition-` string the keyword, that allows the *system* to 
> > > > use
> > > > it when it's a partitioned topic.
> > > >
> > > >
> > > >
> > > > > > > > > > Why postfix of `__`?Why uppercase ?Maybe `__system__`?
> > > > > > Yes, That is a key point that I want to discuss in this
> > > > > > thread. `__system__` is good for me.
> > > > > > > > > > Can you please elaborate what it means to make it dynamic 
> > > > > > > > > > exactly?
> > > > > > Sorry, I will refine it. it means we can update this configuration
> > > > > > dynamically. (using rest api or sth)
> > > > > >
> > > >
> > > > I'm unfamiliar with how Pulsar supports dynamic configuration. I would
> > > > love it if you can share a link or explain it briefly, thus explaining 
> > > > what
> > > > exactly you are going to change to support dynamic configuration.
> > > >
> > > > > > General question: In the last thread you said something about
> > > > > > configurablerules, etc? You decided not to use this idea?
> > > > > > IMO, That idea is an advanced feature. we may need more time to 
> > > > > > discuss
> > > > > > the details and for the topic name restriction, maybe we don't have 
> > > > > > strong
> > > > > > reason to use that.
> > > > > >
> > > > > > We can introduce this advanced feature when we have a need for it.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > >
> > > > I agree. Future PIP and discussion.
> > > >
> > > >
> > > > > >
> > > > > > Best,
> > > > > > Mattison
> > > > > > On Feb 13, 2023, 22:21 +0800, Asaf Mesika , 
> > > > > > wrote:
> > > > > > > > > > You mean that allows the *system* to use it when it's a 
> > > > > > > > > > partitioned
> > > > > > topic?
> > > > > >

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

Hi Asaf

There is a link to introduce the dynamic configuration.
https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration

Best,
Mattison
On Feb 14, 2023, 17:06 +0800, Asaf Mesika , wrote:
> On Tue, Feb 14, 2023 at 3:46 AM  wrote:
>
> > Hi, Asaf
> >
> > Welcome to join this discussion.
> > > > You mean that allows the *system* to use it when it's a partitioned
> > topic?
> > Sorry, I didn't get your point. What do you mean by *system*?
> >
>
> This sentence was a reply to:
>
> 2. Make the `-partition-` string the keyword. That allows the user to use
> > it when it's a partitioned topic.
> >
>
> I wanted to say that this sentence should be:
> Make the `-partition-` string the keyword, that allows the *system* to use
> it when it's a partitioned topic.
>
>
>
> > > > Why postfix of `__`?Why uppercase ?Maybe `__system__`?
> > Yes, That is a key point that I want to discuss in this
> > thread. `__system__` is good for me.
> > > > Can you please elaborate what it means to make it dynamic exactly?
> > Sorry, I will refine it. it means we can update this configuration
> > dynamically. (using rest api or sth)
> >
>
> I'm unfamiliar with how Pulsar supports dynamic configuration. I would
> love it if you can share a link or explain it briefly, thus explaining what
> exactly you are going to change to support dynamic configuration.
>
> > General question: In the last thread you said something about
> > configurablerules, etc? You decided not to use this idea?
> > IMO, That idea is an advanced feature. we may need more time to discuss
> > the details and for the topic name restriction, maybe we don't have strong
> > reason to use that.
> >
> > We can introduce this advanced feature when we have a need for it.
> >
> > WDYT?
> >
>
> I agree. Future PIP and discussion.
>
>
> >
> > Best,
> > Mattison
> > On Feb 13, 2023, 22:21 +0800, Asaf Mesika , wrote:
> > > > You mean that allows the *system* to use it when it's a partitioned
> > topic?
> >

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

Hi, All

First of all, I want to list all of the system topics as follows. That Yunze 
has mentioned before.

Namespace level:

• pulsar/system
• transaction_coordinator_assign
• __transaction_log_
• resource-usage
• pulsar/
• healthcheck

Topic level:

• __change_events
• __transaction_buffer_snapshot_segment
• ...

We can check the details here. [0]

Secondly. I think we need focus on the system topic name prefix. we have some 
options as follows:

• __SYSTEM__
• __system__


Both them make sense for me. As there are already two people prefer `__SYSTEM`. 
if no other people have concern, I will chose it.

Best,
Mattison

-[0] 
https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/naming/SystemTopicNames.java
On Feb 15, 2023, 11:36 +0800, Michael Marshall , wrote:
> I support this PIP, thank you for driving it forward Mattison.
>
> I support using the prefix __SYSTEM__ for system topics, and I agree
> that DLQ and RETRY topics do not qualify as system topics because they
> are produced to and consumed by regular clients.
>
> I think we might benefit from a generic definition for a system topic.
> I provided a loose definition for system topics in this email [0]. I
> copy it here:
>
> A system topic is a topic that is completely internal to Pulsar
> components. Internally, it is a normal topic. It requires elevated
> permission to produce/consume when authorization is enabled, even if
> topic level policies are not enabled. Generic calls like
> `clearNamespaceBacklog` should not affect system topics. Deleting a
> namespace or tenant should delete the system topics within it.
>
> Thanks,
> Michael
>
> [0] https://lists.apache.org/thread/sr01hvqmdrnk4lxwfwzcpg7y21psj6tt
>
>
> On Tue, Feb 14, 2023 at 9:55 AM  wrote:
> >
> > > > Could you list all existing system topic names that are used so we 
> > > > canunderstand the rule better?
> > Yes, sure. I will list it later.
> > > > And I saw that this proposal only forbids the creation of thesetopics. 
> > > > What about writing messages to them? I think it's better notto allow 
> > > > Pulsar clients to write messages to them. The Pulsar clientshould 
> > > > configure something to get the permission to write messages tothem. To 
> > > > keep the compatibility, maybe we can make use of the protocolversion.
> > Sure, I agree with your point, because the current proposal just want to 
> > have topic name restriction. maybe we can revise the system topic 
> > permission and etc in the next proposal. (relate to system topic)
> >
> > Best,
> > Mattison
> > On Feb 14, 2023, 21:22 +0800, Yunze Xu , 
> > wrote:
> > > > Could you list all existing system topic names that are used so we can
> > > > understand the rule better? Such as the "-RETRY" and "-DLQ" topics
> > > > mentioned above.
> > > >
> > > > And I saw that this proposal only forbids the creation of these
> > > > topics. What about writing messages to them? I think it's better not
> > > > to allow Pulsar clients to write messages to them. The Pulsar client
> > > > should configure something to get the permission to write messages to
> > > > them. To keep the compatibility, maybe we can make use of the protocol
> > > > version.
> > > >
> > > > Thanks
> > > > Yunze
> > > >
> > > > On Tue, Feb 14, 2023 at 5:38 PM Yubiao Feng
> > > >  wrote:
> > > > > >
> > > > > > Hi Qiang
> > > > > >
> > > > > > ### System topic name
> > > > > > '__system__' I think this format is clearer.
> > > > > > Now the system automatically creates topics of type retry consumer 
> > > > > > letters
> > > > > > and dead letters.
> > > > > > These topics all end in uppercase letters, such as `-RETRY,` `-DLQ.`
> > > > > > Is it better to define the system topic name in uppercase(
> > > > > > '__SYSTEM__' )?
> > > > > >
> > > > > > ### Another comment
> > > > > > If you now redesign the topic name restrictions, should we make 
> > > > > > `-RETRY`
> > > > > > and `-DLQ` keywords?
> > > > > >
> > > > > > Thanks
> > > > > > Yubiao Feng
> > > > > >
> > > > > > On Tue, Feb 14, 2023 at 5:06 PM Asaf Mesika  
> > > > > > wrote:
> > > > > >
> > > > > > > > > > On Tue, Feb 14, 2023 at 3:46 AM  
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > > > Hi, Asaf
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Welcome to join this discussion.
> > > > > > > > > > > > > > > > > > You mean that allows the *system* to use it 
> > > > > > > > > > > > > > > > > > when it's a partitioned
> > > > > > > > > > > > > > topic?
> > > > > > > > > > > > > > Sorry, I didn't get your point. What do you mean by 
> > > > > > > > > > > > > > *system*?
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This sentence was a reply to:
> > > > > > > > > >
> > > > > > > > > > 2. Make the `-partition-` string the keyword. That allows 
> > > > > > > > > > the user to use
> > > > > > > > > > > > > > it when it's a partitioned topic.
> > > > > > > > > > > > > >
> > > > > > > > > >
> > 

Re: [VOTE][PIP-226] Add JWKS support for AuthenticationProviderToken

[mattisonchao]

+1, (non-binding)

Best,
Mattison
On Dec 22, 2022, 11:24 +0800, Zixuan Liu , wrote:
> Hi all,
>
> Voting for https://github.com/apache/pulsar/issues/18798.
>
> Thanks,
> Zixuan

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

> Could you list all existing system topic names that are used so we 
> canunderstand the rule better?
Yes, sure. I will list it later.
> And I saw that this proposal only forbids the creation of thesetopics. What 
> about writing messages to them? I think it's better notto allow Pulsar 
> clients to write messages to them. The Pulsar clientshould configure 
> something to get the permission to write messages tothem. To keep the 
> compatibility, maybe we can make use of the protocolversion.
Sure,  I agree with your point, because the current proposal just want to have 
topic name restriction. maybe we can revise the system topic permission and etc 
in the next proposal. (relate to system topic)

Best,
Mattison
On Feb 14, 2023, 21:22 +0800, Yunze Xu , wrote:
> Could you list all existing system topic names that are used so we can
> understand the rule better? Such as the "-RETRY" and "-DLQ" topics
> mentioned above.
>
> And I saw that this proposal only forbids the creation of these
> topics. What about writing messages to them? I think it's better not
> to allow Pulsar clients to write messages to them. The Pulsar client
> should configure something to get the permission to write messages to
> them. To keep the compatibility, maybe we can make use of the protocol
> version.
>
> Thanks
> Yunze
>
> On Tue, Feb 14, 2023 at 5:38 PM Yubiao Feng
>  wrote:
> >
> > Hi Qiang
> >
> > ### System topic name
> > '__system__' I think this format is clearer.
> > Now the system automatically creates topics of type retry consumer letters
> > and dead letters.
> > These topics all end in uppercase letters, such as `-RETRY,` `-DLQ.`
> > Is it better to define the system topic name in uppercase(
> > '__SYSTEM__' )?
> >
> > ### Another comment
> > If you now redesign the topic name restrictions, should we make `-RETRY`
> > and `-DLQ` keywords?
> >
> > Thanks
> > Yubiao Feng
> >
> > On Tue, Feb 14, 2023 at 5:06 PM Asaf Mesika  wrote:
> >
> > > > On Tue, Feb 14, 2023 at 3:46 AM  wrote:
> > > >
> > > > > > Hi, Asaf
> > > > > >
> > > > > > Welcome to join this discussion.
> > > > > > > > You mean that allows the *system* to use it when it's a 
> > > > > > > > partitioned
> > > > > > topic?
> > > > > > Sorry, I didn't get your point. What do you mean by *system*?
> > > > > >
> > > >
> > > > This sentence was a reply to:
> > > >
> > > > 2. Make the `-partition-` string the keyword. That allows the user to 
> > > > use
> > > > > > it when it's a partitioned topic.
> > > > > >
> > > >
> > > > I wanted to say that this sentence should be:
> > > > Make the `-partition-` string the keyword, that allows the *system* to 
> > > > use
> > > > it when it's a partitioned topic.
> > > >
> > > >
> > > >
> > > > > > > > Why postfix of `__`?Why uppercase ?Maybe `__system__`?
> > > > > > Yes, That is a key point that I want to discuss in this
> > > > > > thread. `__system__` is good for me.
> > > > > > > > Can you please elaborate what it means to make it dynamic 
> > > > > > > > exactly?
> > > > > > Sorry, I will refine it. it means we can update this configuration
> > > > > > dynamically. (using rest api or sth)
> > > > > >
> > > >
> > > > I'm unfamiliar with how Pulsar supports dynamic configuration. I would
> > > > love it if you can share a link or explain it briefly, thus explaining 
> > > > what
> > > > exactly you are going to change to support dynamic configuration.
> > > >
> > > > > > General question: In the last thread you said something about
> > > > > > configurablerules, etc? You decided not to use this idea?
> > > > > > IMO, That idea is an advanced feature. we may need more time to 
> > > > > > discuss
> > > > > > the details and for the topic name restriction, maybe we don't have
> > > > strong
> > > > > > reason to use that.
> > > > > >
> > > > > > We can introduce this advanced feature when we have a need for it.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > >
> > > > I agree. Future PIP and discussion.
> > > >
> > > >
> > > > > >
> > > > > > Best,
> > > > > > Mattison
> > > > > > On Feb 13, 2023, 22:21 +0800, Asaf Mesika ,
> > > > wrote:
> > > > > > > > You mean that allows the *system* to use it when it's a 
> > > > > > > > partitioned
> > > > > > topic?
> > > > > >
> > > >

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

>These topics all end in uppercase letters, such as `-RETRY,` `-DLQ.`
Is it better to define the system topic name in uppercase(
'__SYSTEM__' )?

I think we don't need to mark `-DLQ` and `-RETRY` to be the keyword. because 
they are the same as the normal topic for the broker side.

Best,
Mattison
On Feb 14, 2023, 17:38 +0800, Yubiao Feng 
, wrote:
> Hi Qiang
>
> ### System topic name
> '__system__' I think this format is clearer.
> Now the system automatically creates topics of type retry consumer letters
> and dead letters.
> These topics all end in uppercase letters, such as `-RETRY,` `-DLQ.`
> Is it better to define the system topic name in uppercase(
> '__SYSTEM__' )?
>
> ### Another comment
> If you now redesign the topic name restrictions, should we make `-RETRY`
> and `-DLQ` keywords?
>
> Thanks
> Yubiao Feng
>
> On Tue, Feb 14, 2023 at 5:06 PM Asaf Mesika  wrote:
>
> > On Tue, Feb 14, 2023 at 3:46 AM  wrote:
> >
> > > > Hi, Asaf
> > > >
> > > > Welcome to join this discussion.
> > > > > > You mean that allows the *system* to use it when it's a partitioned
> > > > topic?
> > > > Sorry, I didn't get your point. What do you mean by *system*?
> > > >
> >
> > This sentence was a reply to:
> >
> > 2. Make the `-partition-` string the keyword. That allows the user to use
> > > > it when it's a partitioned topic.
> > > >
> >
> > I wanted to say that this sentence should be:
> > Make the `-partition-` string the keyword, that allows the *system* to use
> > it when it's a partitioned topic.
> >
> >
> >
> > > > > > Why postfix of `__`?Why uppercase ?Maybe `__system__`?
> > > > Yes, That is a key point that I want to discuss in this
> > > > thread. `__system__` is good for me.
> > > > > > Can you please elaborate what it means to make it dynamic exactly?
> > > > Sorry, I will refine it. it means we can update this configuration
> > > > dynamically. (using rest api or sth)
> > > >
> >
> > I'm unfamiliar with how Pulsar supports dynamic configuration. I would
> > love it if you can share a link or explain it briefly, thus explaining what
> > exactly you are going to change to support dynamic configuration.
> >
> > > > General question: In the last thread you said something about
> > > > configurablerules, etc? You decided not to use this idea?
> > > > IMO, That idea is an advanced feature. we may need more time to discuss
> > > > the details and for the topic name restriction, maybe we don't have
> > strong
> > > > reason to use that.
> > > >
> > > > We can introduce this advanced feature when we have a need for it.
> > > >
> > > > WDYT?
> > > >
> >
> > I agree. Future PIP and discussion.
> >
> >
> > > >
> > > > Best,
> > > > Mattison
> > > > On Feb 13, 2023, 22:21 +0800, Asaf Mesika ,
> > wrote:
> > > > > > You mean that allows the *system* to use it when it's a partitioned
> > > > topic?
> > > >
> >

Re: [VOTE] PIP-175: Extend time based release process

[mattisonchao]

+1(non-binding)

Best,
Mattison
On Feb 9, 2023, 04:44 +0800, Matteo Merli , wrote:
> https://github.com/apache/pulsar/issues/15966
>
> 
>
>
> ## Motivation
>
> In PIP-47 (
> https://github.com/apache/pulsar/wiki/PIP-47:-Time-Based-Release-Plan), we
> have adopted a time-based release plan. This was the first attempt at
> establishing a new principle on how releases should b
>
> The main two benefits of this approach have been:
>
> 1. Clarity for users and developers on when to expect a release
> 2. Breaking a hard relationship between feature and release: a particular
> feature will be included in the release if it is completed in time.
> Otherwise, it will be bubbled up to the next release.
>
> The motivation for the current proposal is to extend the existing process
> to address the issues that we have seen and that were left out of the scope
> of PIP-47.
>
> ## Summary of existing issues in the process
>
> ### Short maintenance cycles for releases
>
> Since we're doing a 3 months release cycle, we are ending with 4 releases
> done per year, even though it's more close to 3 releases.
>
> There is a high cost to maintain a lot of old releases, backport bug fixes,
> and security patches. In general, we actively support the last 3 minor
> releases while continuing to develop the next release. E.g., 2.8, 2.9, and
> 2.10, while 2.11 is under development.
>
> The result is that a user adopting a particular release is forced to
> upgrade in a < 1-year timeframe to keep up to date and use a supported
> release. This timeframe is too short for many users as it imposes a lot of
> forced upgrades, for which they are not prepared in terms of available time
> and required effort.
>
> ### Live Upgrade/Downgrade compatibility path
>
> In Pulsar, we guarantee that users have a way to do live upgrades and
> downgrades with zero downtime.
>
> This is very powerful because it gives them the freedom to upgrade to a new
> release with the assurance of being able to roll back to the previous
> release in case any functional or performance regressions are encountered.
>
> Today, this compatibility is guaranteed across minor versions. Eg: I can do
> `2.7 -> 2.8 -> 2.7` as a live upgrade.
>
> What is not guaranteed is to "skip" releases. E.g.: `2.7 -> 2.9` might work
> or not, but it's not guaranteed. In that case an intermediated upgrade
> would be required: `2.7 -> 2.8 -> 2.9`.
>
> The reasons for which the "skip" upgrade might not work are multiple:
> 1. Incompatible upgrade of some dependency (e.g., ZooKeeper) that might
> not be compatible with an older version.
> 2. Adoption of a new metadata format or data format on disk.
> Every time we introduce a new incompatible format change (outside of a
> regular Protobuf field addition), we do it in a 2 steps way:
> - In a new release, we introduce the new feature/format, disabled by
> default. The new release can read both old and new formats, though it keeps
> writing the old format by default.
> - In a subsequent release, we change the default to the new format
>
> Note that this consideration is separate from the compatibility between
> clients and brokers, where we ***never*** break compatibility. The oldest
> available Pulsar client can still talk with the newest Pulsar broker, and
> vice versa, a new client, will be perfectly fine with an older broker
> (except the new features won't be working).
>
> ### Releases getting delayed
>
> Another problem we have been experiencing is that release cycles have been
> stretching considerably. Part of this has been because we have been
> reaching the end of the release window, preparing a candidate, and then
> taking a long time to flush out all issues found at the last minute in the
> new release.
>
> We need to ensure that we have a date set in stone to deliver the release
> to users.
>
> ## Proposal
>
> The proposal to address the above issues is composed of 2 parts.
>
> ### 1. Establish Long Term Support releases
>
> We need to provide a way for users to quickly understand the expected
> lifecycle timeline of a given release and for that timeline to be long
> enough not to be a constant update mandate.
>
> At the same time, we need to ensure that we maintainers are not spending
> all the time just maintaining a huge list of old releases.
>
> For that, we can use the established concept of "Long Term Releases" or
> LTS.
>
> We will perform LTS releases at a fixed cadence every 18 months, and we
> will keep doing regular feature releases every 3 months as we're currently
> doing.
>
> The LTS releases will be identified by being a `.0` version. For example:
> * `3.0` -> LTS
> * `3.1` -> regular release
> * `3.2` -> regular release
> * `4.0` -> LTS
>
> The major version bump will not carry any special meaning in terms of "big
> features" included in the release or breaking API changes. Instead, it
> would simply signal the type of the release.
>
> 

Re: [DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

Hi, Asaf

Welcome to join this discussion.
> You mean that allows the *system* to use it when it's a partitioned topic?
Sorry, I didn't get your point. What do you mean by *system*?
> Why postfix of `__`?Why uppercase ?Maybe `__system__`?
Yes, That is a key point that I want to discuss in this thread.  
`__system__` is good for me.
> Can you please elaborate what it means to make it dynamic exactly?
Sorry, I will refine it. it means we can update this configuration dynamically. 
(using rest api or sth)
> General question: In the last thread you said something about 
> configurablerules, etc? You decided not to use this idea?
IMO, That idea is an advanced feature. we may need more time to discuss the 
details and for the topic name restriction, maybe we don't have strong reason 
to use that.

We can introduce this advanced feature when we have a need for it.

WDYT?

Best,
Mattison
On Feb 13, 2023, 22:21 +0800, Asaf Mesika , wrote:
> You mean that allows the *system* to use it when it's a partitioned topic?

[DISCUSS] PIP-242 Topic name restrictions

[mattisonchao]

Hi guys.

This is the refined PIP-242 discussion thread. Because we have discussed some 
parts of this in another thread[0][1], we can start the standard PIP discussion 
to reach a detailed consensus.

You can check here[2] to see the PIP.


Best,
Mattison

[0] https://lists.apache.org/thread/5s4kop7qmxgpvsnh65s42r9mv7qc1pxt
[1] https://lists.apache.org/thread/km7jp26zmnfj6t3hpq0rt77j7wvn45o6
[2] https://github.com/apache/pulsar/issues/19239

-- Paste the document to help quotes --

### Motivation

Currently, the Apache Pulsar has no restrictions on the topic name. Even 
`system topics`, `partitioned topics` etc. which is confusing for our 
developers to identify whether this is a partition of a partitioned topic. 
Plus, we need to add more logic to be compatible with this special topic name. 
for example:

- https://github.com/apache/pulsar/pull/19240
- https://github.com/apache/pulsar/pull/19230
- https://github.com/apache/pulsar/pull/19171
- https://github.com/apache/pulsar/pull/19086
- ...

### Goal

This proposal aimed to add more restrictions for a topic name. that includes 
three parts as follows:
1. Using `NamedEntity` to validate the topic name characters.
2. Make the `-partition-` string the keyword. That allows the user to use it 
when it's a partitioned topic.
3. Introduce the new system topic structure. `__SYS`


**Create a topic:**
_no corresponding partitioned topic_
- persistent://public/default/local-name (passed)
- persistent://public/default/local-name-partition-z (rejected by keyword)
- persistent://public/default/local-name-partition-0 (rejected by keyword)

_Has corresponding partitioned topic, **partitions=2** and topic partition name 
is **persistent://public/default/local-name**_

- persistent://public/default/local-name-partition-0 (passed, Because it is the 
partition topic's sub-partition)
- persistent://public/default/local-name-partition-z (rejected by keyword)
- persistent://public/default/local-name-partition-4 (rejected, Because it 
exceeds the number of maximum partitions)

**Create a partitioned topic(topic metadata)**

- persistent://public/default/local-name (passed)
- persistent://public/default/local-name-partition-z (rejected by keyword)
- persistent://public/default/local-name-partition-0 (rejected by keyword)

### Compatibility

We will introduce the new configuration `enableStrictTopicName`  to support 
compatibility. Users can easily disable it to get rid of the restrictions.

For the system topic names, I would like to keep the original names and 
introduce a new topic structure `__SYS`

### API Changes

Add a new configuration, `enableStrictTopicName=false`.

### Implementation

1. Add configuration `enableStrictTopicName=false` and make it dynamic.
2. Add topic name validation by `NamedEntity`
3. Add partitioned topic rejection logic.
4. Add warning logs to inform users that we do not recommend creating topics 
that name with the restrictions name when users disable `enableStrictTopicName`.
5. Make `enableStrictTopicName=true` in the next major release. and mark that 
this config will be removed in the future.

Re: [DISCUSS] Topic name restriction

[mattisonchao]

Hi guys.

Thanks for your discussion in this thread. Since we have reached the discussion 
deadline.
> I will keep this discussion for a week. If there are no more new types of 
> restrictions, I will refine the previous PIP-242[0] to explain more details.

I would like to refine the PIP-242, which includes four parts.

• Introduce `enableStrictTopicName.` configuration.
• Add NamedEntity validation for the topic name.
• Add `-partition-` keyword.
• Introduce topic name's structure.（we can keep the original topic names and 
introduce new system topic name structure `__SYS`. ）


Additional:
Since the disallowed topic names configuration needs more discussion about name 
pattern, type etc. I think we can wait for the demand to consider it.

Best,
Mattison

On Feb 6, 2023, 23:10 +0800, mattisonc...@gmail.com, wrote:
> Hi, Asaf
> > I don't understand the idea suggested of making the validation 
> > rulesconfigurable.If understand correctly:* "-partition" is not something 
> > you want configurable - it should always bevalidated* System topics - once 
> > we agree on a naming convention going forward, itshould always be validated.
> We need to ensure compatibility so that users can choose. as Michael 
> mentioned. the configurable restriction can easily help users to avoid 
> breaking.
> > In the context of PIP 242, we're introducing a config to optionallyenforce 
> > strict topic names. As such, we could rely on the config toeither use the 
> > "cheap" check to see if the topic starts with __ or wecould use the more 
> > expensive check to determine if the topic name isone of many possible 
> > system topic names. Because we want to maintainbackwards compatibility, we 
> > cannot completely get rid of the oldlogic.
> Best,
> Mattison
> On Feb 5, 2023, 19:24 +0800, Asaf Mesika , wrote:
> > Thanks Mattison.
> >
> > I don't understand the idea suggested of making the validation rules
> > configurable.
> > If understand correctly:
> > * "-partition" is not something you want configurable - it should always be
> > validated
> > * System topics - once we agree on a naming convention going forward, it
> > should always be validated.
> >
> > I'm ok with adding configuration for the user so they can validate rules of
> > their own, maybe even per tenant.
> >
> >
> > On Fri, Feb 3, 2023 at 11:44 AM  wrote:
> >
> > > Hi, Asaf
> > >
> > > We are using the regular expression to check the name.
> > > "^[-=:.\\w]*$"
> > > The \w means [A-Za-z0-9_] , which includes underscores.
> > >
> > > Best,
> > > Mattison
> > > On Feb 2, 2023, 23:01 +0800, Asaf Mesika , wrote:
> > > > > NamedEntity is not allowing underscores - does it make sense?
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 2, 2023 at 8:35 AM Michael Marshall 
> > > > > wrote:
> > > > >
> > > > > > > Thanks for starting this thread, Mattison.
> > > > > > >
> > > > > > > > > > > The topic name character validation is already done by
> > > > > > > > > > > `NamedEntity#checkName`.
> > > > > > >
> > > > > > > Based on my reading of the code, only the tenant and the namespace
> > > > > > > names are validated using that method. There is a call [0] that 
> > > > > > > looks
> > > > > > > like it validates topic names, but that method is only called by
> > > > > > > tests.
> > > > > > >
> > > > > > > > > > > But I have a concern that whether we should
> > > > > > > > > > > treat all topics that start with the long underscore 
> > > > > > > > > > > ("__") as
> > > system
> > > > > > > > > > > topics?
> > > > > > >
> > > > > > > This is a reasonable concern, and my primary motivation in 
> > > > > > > proposing
> > > > > > > this change is to make it easier for the broker to handle system
> > > > > > > topics, which often get unique treatment.
> > > > > > >
> > > > > > > I wrote on this topic in several replies on this thread from a 
> > > > > > > year ago
> > > > > > > [1].
> > > > > > >
> > > > > > > In the context of PIP 242, we're introducing a config to 
> > > > > > > optionally
> > > > > > > enforce strict topic names. As such, we could rely on the config 
> > > > > > > to
> > > > > > > either use the "cheap" check to see if the topic starts with __ 
> > > > > > > or we
> > > > > > > could use the more expensive check to determine if the topic name 
> > > > > > > is
> > > > > > > one of many possible system topic names. Because we want to 
> > > > > > > maintain
> > > > > > > backwards compatibility, we cannot completely get rid of the old
> > > > > > > logic. I like self describing names because they are elegant and
> > > > > > > efficient.
> > > > > > >
> > > > > > > > > > > If yes, how would you like to allow users to access the 
> > > > > > > > > > > system
> > > topics?
> > > > > > >
> > > > > > > I proposed some ideas at the end of that thread [1]. We should 
> > > > > > > have a
> > > > > > > clear definition of system topics and how they are or are not 
> > > > > > > accessed
> > > by
> > > > > > > users. Ultimately, we continue to create new system 

Re: [DISCUSS] Topic name restriction

[mattisonchao]

Hi, Asaf
> I don't understand the idea suggested of making the validation 
> rulesconfigurable.If understand correctly:* "-partition" is not something you 
> want configurable - it should always bevalidated* System topics - once we 
> agree on a naming convention going forward, itshould always be validated.
We need to ensure compatibility so that users can choose. as Michael mentioned. 
the configurable restriction can easily help users to avoid breaking.
> In the context of PIP 242, we're introducing a config to optionallyenforce 
> strict topic names. As such, we could rely on the config toeither use the 
> "cheap" check to see if the topic starts with __ or wecould use the more 
> expensive check to determine if the topic name isone of many possible system 
> topic names. Because we want to maintainbackwards compatibility, we cannot 
> completely get rid of the oldlogic.
Best,
Mattison
On Feb 5, 2023, 19:24 +0800, Asaf Mesika , wrote:
> Thanks Mattison.
>
> I don't understand the idea suggested of making the validation rules
> configurable.
> If understand correctly:
> * "-partition" is not something you want configurable - it should always be
> validated
> * System topics - once we agree on a naming convention going forward, it
> should always be validated.
>
> I'm ok with adding configuration for the user so they can validate rules of
> their own, maybe even per tenant.
>
>
> On Fri, Feb 3, 2023 at 11:44 AM  wrote:
>
> > Hi, Asaf
> >
> > We are using the regular expression to check the name.
> > "^[-=:.\\w]*$"
> > The \w means [A-Za-z0-9_] , which includes underscores.
> >
> > Best,
> > Mattison
> > On Feb 2, 2023, 23:01 +0800, Asaf Mesika , wrote:
> > > > NamedEntity is not allowing underscores - does it make sense?
> > > >
> > > >
> > > >
> > > > On Thu, Feb 2, 2023 at 8:35 AM Michael Marshall 
> > > > wrote:
> > > >
> > > > > > Thanks for starting this thread, Mattison.
> > > > > >
> > > > > > > > > > The topic name character validation is already done by
> > > > > > > > > > `NamedEntity#checkName`.
> > > > > >
> > > > > > Based on my reading of the code, only the tenant and the namespace
> > > > > > names are validated using that method. There is a call [0] that 
> > > > > > looks
> > > > > > like it validates topic names, but that method is only called by
> > > > > > tests.
> > > > > >
> > > > > > > > > > But I have a concern that whether we should
> > > > > > > > > > treat all topics that start with the long underscore ("__") 
> > > > > > > > > > as
> > system
> > > > > > > > > > topics?
> > > > > >
> > > > > > This is a reasonable concern, and my primary motivation in proposing
> > > > > > this change is to make it easier for the broker to handle system
> > > > > > topics, which often get unique treatment.
> > > > > >
> > > > > > I wrote on this topic in several replies on this thread from a year 
> > > > > > ago
> > > > > > [1].
> > > > > >
> > > > > > In the context of PIP 242, we're introducing a config to optionally
> > > > > > enforce strict topic names. As such, we could rely on the config to
> > > > > > either use the "cheap" check to see if the topic starts with __ or 
> > > > > > we
> > > > > > could use the more expensive check to determine if the topic name is
> > > > > > one of many possible system topic names. Because we want to maintain
> > > > > > backwards compatibility, we cannot completely get rid of the old
> > > > > > logic. I like self describing names because they are elegant and
> > > > > > efficient.
> > > > > >
> > > > > > > > > > If yes, how would you like to allow users to access the 
> > > > > > > > > > system
> > topics?
> > > > > >
> > > > > > I proposed some ideas at the end of that thread [1]. We should have 
> > > > > > a
> > > > > > clear definition of system topics and how they are or are not 
> > > > > > accessed
> > by
> > > > > > users. Ultimately, we continue to create new system topics without
> > > > > > reserving a designated naming structure and without defining how 
> > > > > > these
> > > > > > topics ought to be interacted with, as Yunze points out. Note that 
> > > > > > any
> > > > > > system topic we introduce could conflict with existing user topics, 
> > > > > > so
> > > > > > proactively reserving a set of names makes it easier for forwards
> > > > > > compatibility.
> > > > > >
> > > > > > Thanks,
> > > > > > Michael
> > > > > >
> > > > > > [0]
> > > > > >
> > https://github.com/apache/pulsar/blob/b880b1d240ade864181935aa360bfca03a5aa67f/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamespaceName.java#L159
> > > > > > [1] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 1, 2023 at 11:28 PM r...@apache.org <
> > ranxiaolong...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mattison:
> > > > > > > > > >
> > > > > > > > > > Agree with Yong's idea. We can expose `disallowed topic` as 
> > > > > > > > > > a
> > > > > > configuration
> > > > > > 

Re: [DISCUSS] Topic name restriction

[mattisonchao]

Hi, Asaf

We are using the regular expression to check the name.
"^[-=:.\\w]*$"
The \w means [A-Za-z0-9_] , which includes underscores.

Best,
Mattison
On Feb 2, 2023, 23:01 +0800, Asaf Mesika , wrote:
> NamedEntity is not allowing underscores - does it make sense?
>
>
>
> On Thu, Feb 2, 2023 at 8:35 AM Michael Marshall 
> wrote:
>
> > Thanks for starting this thread, Mattison.
> >
> > > > The topic name character validation is already done by
> > > > `NamedEntity#checkName`.
> >
> > Based on my reading of the code, only the tenant and the namespace
> > names are validated using that method. There is a call [0] that looks
> > like it validates topic names, but that method is only called by
> > tests.
> >
> > > > But I have a concern that whether we should
> > > > treat all topics that start with the long underscore ("__") as system
> > > > topics?
> >
> > This is a reasonable concern, and my primary motivation in proposing
> > this change is to make it easier for the broker to handle system
> > topics, which often get unique treatment.
> >
> > I wrote on this topic in several replies on this thread from a year ago
> > [1].
> >
> > In the context of PIP 242, we're introducing a config to optionally
> > enforce strict topic names. As such, we could rely on the config to
> > either use the "cheap" check to see if the topic starts with __ or we
> > could use the more expensive check to determine if the topic name is
> > one of many possible system topic names. Because we want to maintain
> > backwards compatibility, we cannot completely get rid of the old
> > logic. I like self describing names because they are elegant and
> > efficient.
> >
> > > > If yes, how would you like to allow users to access the system topics?
> >
> > I proposed some ideas at the end of that thread [1]. We should have a
> > clear definition of system topics and how they are or are not accessed by
> > users. Ultimately, we continue to create new system topics without
> > reserving a designated naming structure and without defining how these
> > topics ought to be interacted with, as Yunze points out. Note that any
> > system topic we introduce could conflict with existing user topics, so
> > proactively reserving a set of names makes it easier for forwards
> > compatibility.
> >
> > Thanks,
> > Michael
> >
> > [0]
> > https://github.com/apache/pulsar/blob/b880b1d240ade864181935aa360bfca03a5aa67f/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamespaceName.java#L159
> > [1] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> >
> >
> > On Wed, Feb 1, 2023 at 11:28 PM r...@apache.org 
> > wrote:
> > > >
> > > > Hi Mattison:
> > > >
> > > > Agree with Yong's idea. We can expose `disallowed topic` as a
> > configuration
> > > > to the user side, and a more flexible way is to expose it as a
> > > > namespace-level policy. This can ensure that there is no need to do
> > special
> > > > processing on customized keywords in the future, and the expected effect
> > > > can be achieved by modifying the configuration.
> > > >
> > > > Think Yunze's concerns are justified for the system topic. Is it okay if
> > we
> > > > use hard code? Because the identification of any keyword is likely to be
> > > > hit by the user. The hard code method is used to filter out system 
> > > > topics
> > > > and not allow users to operate during delete and create operations.
> > > >
> > > > --
> > > > Thanks
> > > > Xiaolong Ran
> > > >
> > > > Dave Fisher  于2023年2月2日周四 11:26写道：
> > > >
> > > > > >
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > > On Feb 1, 2023, at 6:52 PM, Yong Zhang 
> > > > > > > > 
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Mattison,
> > > > > > > >
> > > > > > > > I agree with you about restricting the topic name.
> > > > > > > >
> > > > > > > > How about using a blacklist way to restrict it?
> > > > > >
> > > > > > If we do then please call it by another name like “disallowed”.
> > > > > >
> > > > > > > >
> > > > > > > > We can have a blacklist on the topic name restriction and make 
> > > > > > > > it
> > > > > > > > configurable. Add the keywords you mentioned in the default
> > > > > > configuration.
> > > > > > > > That would have a more general way to block a topic name 
> > > > > > > > creation.
> > > > > > > > If we have more restrictions on the topic name in the future, 
> > > > > > > > this
> > way
> > > > > > > > can make it easy to fit them without changing any code.
> > > > > >
> > > > > > Is there anyone asking for this feature?
> > > > > >
> > > > > > Best,
> > > > > > Dave
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yong
> > > > > > > >
> > > > > > > > >> On Thu, 2 Feb 2023 at 07:33,  wrote:
> > > > > > > > >>
> > > > > > > > >> Hi, All
> > > > > > > > >>
> > > > > > > > >> In the current implementation, pulsar didn't support topic 
> > > > > > > > >> name
> > > > > > > > >> restriction. It's a good chance to discuss it.
> > > > > > > > >>
> > > > > > > > >> I 

Re: [DISCUSS] Topic name restriction

[mattisonchao]

Hi, Yunze
> The topic name character validation is already done by`NamedEntity#checkName`
As Michael mentioned, the `NamedEntity#checkName` just checked the tenant and 
namespace.
> But I have a concern that whether we shouldtreat all topics that start with 
> the long underscore ("__") as systemtopics? Users might have defined their 
> own "system topics" that startwith the long underscore for special uses.If 
> yes, how would you like to allow users to access the system topics?In Kafka, 
> there are a few (only two, IIRC) special topics that onlyallow non-admin 
> users to read, which means users cannot write to thesetopics or delete them.
We can add the warn log when the user creates a topic with the `__` start. To 
tell users that we will mark this as the system topic prefix keyword. It will 
probably be banned in the future.
Or we can just add the existent system topic name as the keywords and introduce 
a new complex system topic rule.

Best,
Mattison
On Feb 2, 2023, 11:11 +0800, Yunze Xu , wrote:
> The topic name character validation is already done by
> `NamedEntity#checkName`. And I agree that the system topic should be
> taken carefully as well. But I have a concern that whether we should
> treat all topics that start with the long underscore ("__") as system
> topics? Users might have defined their own "system topics" that start
> with the long underscore for special uses.
>
> If yes, how would you like to allow users to access the system topics?
> In Kafka, there are a few (only two, IIRC) special topics that only
> allow non-admin users to read, which means users cannot write to these
> topics or delete them.
>
> Thanks,
> Yunze
>
> On Thu, Feb 2, 2023 at 10:52 AM Yong Zhang  wrote:
> >
> > Mattison,
> >
> > I agree with you about restricting the topic name.
> >
> > How about using a blacklist way to restrict it?
> >
> > We can have a blacklist on the topic name restriction and make it
> > configurable. Add the keywords you mentioned in the default configuration.
> > That would have a more general way to block a topic name creation.
> > If we have more restrictions on the topic name in the future, this way
> > can make it easy to fit them without changing any code.
> >
> > Thanks,
> > Yong
> >
> > On Thu, 2 Feb 2023 at 07:33,  wrote:
> >
> > > > Hi, All
> > > >
> > > > In the current implementation, pulsar didn't support topic name
> > > > restriction. It's a good chance to discuss it.
> > > >
> > > > I think this discussion aims to identify what types of topic names we 
> > > > all
> > > > need to restrict.
> > > >
> > > > I know three topic names that need to be restricted at the moment.
> > > >
> > > > 1. The `-partition-` keyword.
> > > > 2. Topic name characters validation.
> > > > 3. System topic prefix `__`.
> > > >
> > > >
> > > > Please feel free to leave your comments.
> > > > I will keep this discussion for a week. If there are no more new types 
> > > > of
> > > > restrictions, I will refine the previous PIP-242[0] to explain more 
> > > > details.
> > > > > > If we have other restrictions behind this discussion. We can draft 
> > > > > > a new
> > > > PIP to add it directly.
> > > > Thanks to Michael's opinion[1], we can expand the PIP-242 scopes to help
> > > > pulsar have a good topic name restriction.
> > > >
> > > > Best,
> > > > Mattison
> > > >
> > > > [0] https://github.com/apache/pulsar/issues/19239
> > > > [1] https://lists.apache.org/thread/dd1kxhodjvovtb8yyojkk209st4o0ft2
> > > >

Re: [DISCUSS] Topic name restriction

[mattisonchao]

Hi, Yong

> How about using a blacklist way to restrict it?

It's a great idea. Maybe we can define the rules like  `keyword` with regular 
expressions.

Best,
Mattison


On Feb 2, 2023, 10:52 +0800, Yong Zhang , wrote:
> Mattison,
>
> I agree with you about restricting the topic name.
>
> How about using a blacklist way to restrict it?
>
> We can have a blacklist on the topic name restriction and make it
> configurable. Add the keywords you mentioned in the default configuration.
> That would have a more general way to block a topic name creation.
> If we have more restrictions on the topic name in the future, this way
> can make it easy to fit them without changing any code.
>
> Thanks,
> Yong
>
> On Thu, 2 Feb 2023 at 07:33,  wrote:
>
> > Hi, All
> >
> > In the current implementation, pulsar didn't support topic name
> > restriction. It's a good chance to discuss it.
> >
> > I think this discussion aims to identify what types of topic names we all
> > need to restrict.
> >
> > I know three topic names that need to be restricted at the moment.
> >
> > 1. The `-partition-` keyword.
> > 2. Topic name characters validation.
> > 3. System topic prefix `__`.
> >
> >
> > Please feel free to leave your comments.
> > I will keep this discussion for a week. If there are no more new types of
> > restrictions, I will refine the previous PIP-242[0] to explain more details.
> > > > If we have other restrictions behind this discussion. We can draft a new
> > PIP to add it directly.
> > Thanks to Michael's opinion[1], we can expand the PIP-242 scopes to help
> > pulsar have a good topic name restriction.
> >
> > Best,
> > Mattison
> >
> > [0] https://github.com/apache/pulsar/issues/19239
> > [1] https://lists.apache.org/thread/dd1kxhodjvovtb8yyojkk209st4o0ft2
> >

[DISCUSS] Topic name restriction

[mattisonchao]

Hi, All

In the current implementation, pulsar didn't support topic name restriction. 
It's a good chance to discuss it.

I think this discussion aims to identify what types of topic names we all need 
to restrict.

I know three topic names that need to be restricted at the moment.

1. The `-partition-` keyword.
2. Topic name characters validation.
3. System topic prefix `__`.


Please feel free to leave your comments.
I will keep this discussion for a week. If there are no more new types of 
restrictions, I will refine the previous PIP-242[0] to explain more details.
> If we have other restrictions behind this discussion. We can draft a new PIP 
> to add it directly.
Thanks to Michael's opinion[1], we can expand the PIP-242 scopes to help pulsar 
have a good topic name restriction.

Best,
Mattison

[0] https://github.com/apache/pulsar/issues/19239
[1] https://lists.apache.org/thread/dd1kxhodjvovtb8yyojkk209st4o0ft2

Re: [Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.

[mattisonchao]

Passed the voting by  8 +1 (5 binding and 3 non-binding)

Closed.

Best,
Mattison
On Jan 31, 2023, 06:57 +0800, mattisonc...@gmail.com, wrote:
> Hello everyone.
>
> I would like to start the vote for PIP-242 
> https://github.com/apache/pulsar/issues/19239,
> Please let me know if you have any concerns or questions.
>
> Best,
> Mattison
>
> --- Paste original PIP content to help quote --
>
> ### Motivation
>
> Currently, the Apache Pulsar broker allows users to create a topic name that 
> includes `-partition-`, which is confusing for our developers to identify 
> whether this is a partition of a partitioned topic. Plus, we need to add more 
> logic to be compatible with this special topic name. for example:
>
> - https://github.com/apache/pulsar/pull/19240
> - https://github.com/apache/pulsar/pull/19230
> - https://github.com/apache/pulsar/pull/19171
> - https://github.com/apache/pulsar/pull/19086
> - ...
>
> ### Goal
> This proposal wants `-partition-` to be a topic name keyword. Users can only 
> create a topic with it if the topic is partitioned. For the compatibility 
> reason, we want to Introduce a new configuration - `enableStrictTopicName` 
> for the broker to help reject creating a topic in the following cases:
> 1. Create a partitioned topic that includes `-partition-`.
> 2. Create a topic which is not a partitioned topic.
>
> **Create a topic:**
> _no corresponding partitioned topic_
>
> - persistent://public/default/local-name (passed)
> - persistent://public/default/local-name-partition-z (rejected by keyword)
> - persistent://public/default/local-name-partition-0 (rejected by keyword)
>
> _Has corresponding partitioned topic, **partitions=2** and topic partition 
> name is **persistent://public/default/local-name**_
>
> - persistent://public/default/local-name-partition-0 (passed, Because it is 
> the partition topic's sub-partition)
> - persistent://public/default/local-name-partition-z (rejected by keyword)
> - persistent://public/default/local-name-partition-4 (rejected, Because it 
> exceeds the number of maximum partitions)
>
> **Create a partitioned topic(topic metadata)**
>
> - persistent://public/default/local-name (passed)
> - persistent://public/default/local-name-partition-z (rejected by keyword)
> - persistent://public/default/local-name-partition-0 (rejected by keyword)
>
>
> ### API Changes
>
> Add a new configuration, `enableStrictTopicName=false`.
>
> ### Implementation
>
> 1. Add configuration `enableStrictTopicName=false`.
> 2. Add rejection logic when the user enables `enableStrictTopicName`.
> 4. Add warning logs to inform users that we do not recommend creating 
> non-partitioned topics with the keyword `-partition-`.
> 5. Make `enableStrictTopicName=true` in the next major release.

[Vote] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.

[mattisonchao]

Hello everyone.

I would like to start the vote for PIP-242 
https://github.com/apache/pulsar/issues/19239,
Please let me know if you have any concerns or questions.

Best,
Mattison

--- Paste original PIP content to help quote --

### Motivation

Currently, the Apache Pulsar broker allows users to create a topic name that 
includes `-partition-`, which is confusing for our developers to identify 
whether this is a partition of a partitioned topic. Plus, we need to add more 
logic to be compatible with this special topic name. for example:

- https://github.com/apache/pulsar/pull/19240
- https://github.com/apache/pulsar/pull/19230
- https://github.com/apache/pulsar/pull/19171
- https://github.com/apache/pulsar/pull/19086
- ...

### Goal
This proposal wants `-partition-` to be a topic name keyword. Users can only 
create a topic with it if the topic is partitioned. For the compatibility 
reason, we want to Introduce a new configuration - `enableStrictTopicName` for 
the broker to help reject creating a topic in the following cases:
1. Create a partitioned topic that includes `-partition-`.
2. Create a topic which is not a partitioned topic.

**Create a topic:**
_no corresponding partitioned topic_

- persistent://public/default/local-name (passed)
- persistent://public/default/local-name-partition-z (rejected by keyword)
- persistent://public/default/local-name-partition-0 (rejected by keyword)

_Has corresponding partitioned topic, **partitions=2** and topic partition name 
is **persistent://public/default/local-name**_

- persistent://public/default/local-name-partition-0 (passed, Because it is the 
partition topic's sub-partition)
- persistent://public/default/local-name-partition-z (rejected by keyword)
- persistent://public/default/local-name-partition-4 (rejected, Because it 
exceeds the number of maximum partitions)

**Create a partitioned topic(topic metadata)**

- persistent://public/default/local-name (passed)
- persistent://public/default/local-name-partition-z (rejected by keyword)
- persistent://public/default/local-name-partition-0 (rejected by keyword)


### API Changes

Add a new configuration, `enableStrictTopicName=false`.

### Implementation

1. Add configuration `enableStrictTopicName=false`.
2. Add rejection logic when the user enables `enableStrictTopicName`.
4. Add warning logs to inform users that we do not recommend creating 
non-partitioned topics with the keyword `-partition-`.
5. Make `enableStrictTopicName=true` in the next major release.

[DISCUSS][txn] Move checked exception into builder when newTransaction.

[mattisonchao]

Hello, everyone

I submitted this PR https://github.com/apache/pulsar/pull/19356 to discuss if 
we can accept moving this checked exception into the builder to avoid adding 
more useless try-catch blocks.

In practice, we should create the new transaction like this:

```java
final TransactionBuilder transactionBuilder;
try {
    transactionBuilder = client.newTransaction();
} catch (PulsarClientException e) {
    // handle this exception.
    return;
}
CompletableFuture txnFuture = transactionBuilder
        .withTransactionTimeout(1, TimeUnit.MINUTES)
        .build();
```
But this exception only throws by

```java
 public TransactionBuilder newTransaction() throws PulsarClientException {
    if (!conf.isEnableTransaction()) {
        throw new 
PulsarClientException.InvalidConfigurationException("Transactions are not 
enabled");
    }
    return new TransactionBuilderImpl(this, tcClient);
}
```
Therefore, even if we enabled the transaction, we still need to handle this 
checked exception. Maybe we can move this checked exception into the builder or 
make this exception to be runtime to avoid writing the more useless try-catch 
blocks.

```java
client.newTransaction()
      .withTransactionTimeout(1, TimeUnit.MINUTES)
      .build()
      .thenCompose(transaction -> {
          // do somethings
      }).exceptionally(ex -> {
          // handle exception
      });
```
It is better, isn't it?

Best,
Mattison

Re: [DISCUSS] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.

[mattisonchao]

Hi,  Asaf, Yunze
> You mean to say, if the topic is partitioned, the word "partition" can 
> notappear in the submitted topic name, in the topic creation API?
It's a little bit confusing. I will give some examples to help explain:

Create a topic:

• no corresponding partitioned topic
• persistent://public/default/local-name (passed)
• persistent://public/default/local-name-partition-z (rejected by keyword)
• persistent://public/default/local-name-partition-0 (rejected by keyword)
• Has corresponding partitioned topic, partitions=2 and topic partition name is 
persistent://public/default/local-name
• persistent://public/default/local-name-partition-0 (passed, Because it is 
the partition topic's sub-partition)
• persistent://public/default/local-name-partition-z (rejected by keyword)
• persistent://public/default/local-name-partition-4 (rejected, Because it 
exceeds the number of maximum partitions)


Create a partitioned topic(topic metadata)

• persistent://public/default/local-name (passed)
• persistent://public/default/local-name-partition-z (rejected by keyword)
• persistent://public/default/local-name-partition-0 (rejected by keyword)

> I think this PIP should go close to this end to end, meaning the last 
> stepbeing making it default true.Otherwise, we end up having so many "feature 
> flags" turned off, it's hardto navigate and improve Pulsar.
Yes, we will add a warning log-in step 3 at the current release and enable it 
by default in the next major release(3.0?).

Thanks a lot!

Best,
Mattison
On Jan 30, 2023, 00:17 +0800, Asaf Mesika , wrote:
> You mean to say, if the topic is partitioned, the word "partition" can not 
> appear in the submitted topic name, in the topic creation API?

[DISCUSS] PIP-242: Introduce enableStrictTopicName to reject creating topic with -partition- keyword.

[mattisonchao]

Hello everyone.
I hope you guys are all doing well.

I would like to start the discussion for PIP-242 
https://github.com/apache/pulsar/issues/19239,
Please let me know if you have any concerns or questions.

Best,
Mattison

--- Paste original PIP content to help quote --

### Motivation

Currently, the Apache Pulsar broker allows users to create a topic name that 
includes `-partition-`, which is confusing for our developers to identify 
whether this is a partition of a partitioned topic. Plus, we need to add more 
logic to be compatible with this special topic name. for example:

- https://github.com/apache/pulsar/pull/19240
- https://github.com/apache/pulsar/pull/19230
- https://github.com/apache/pulsar/pull/19171
- https://github.com/apache/pulsar/pull/19086
- ...

### Goal
This proposal wants `-partition-` to be a topic name keyword. Users can only 
create a topic with it if the topic is partitioned. For the compatibility 
reason, we want to Introduce a new configuration - `enableStrictTopicName` for 
the broker to help reject creating a topic in the following cases:
1. Create a partitioned topic that includes `-partition-`.
2. Create a topic which is not a partitioned topic.

### API Changes

Add a new configuration, `enableStrictTopicName=false`.

### Implementation

1. Add configuration `enableStrictTopicName=false`.
2. Add rejection logic when the user enables `enableStrictTopicName`.
3. Add warning logs to inform users that we do not recommend creating 
non-partitioned topics with the keyword `-partition-`.
4. Make `enableStrictTopicName=true` in the future.

Re: [DISCUSS] Code freeze for Pulsar 2.12

[mattisonchao]

Isn't the next version LTS 3.0?

Best
Mattison
On Jan 20, 2023, 07:11 +0800, Christophe Bornet , wrote:
> Hi Pulsar community,
>
> It's great that we released Pulsar 2.11. It has taken quite some time to
> stabilize the release branch and now we have more than 5 months of awesome
> features and commits on the master branch that would benefit a lot to our
> users. That's why I'd like to propose to start a code freeze for the
> release of Pulsar 2.12 with a target release date by mid/end-february.
> Hopefully this release will be easier to stabilize but we don't know for
> sure, so better to start the release activities asap.
> We also need a release manager. Nicolo proposed himself last time but had
> to hand over because of his holiday schedule. So Nicolo, maybe you'd like
> to propose yourself again for this one ? Otherwise I'm happy to volunteer.
>
> Let me know what you think.
>
> Cheers
>
> Christophe

Re: [DISCUSS] PIP-240 A new API to unload subscriptions

[mattisonchao]

Hi, Yubiao

I agree with this idea because some users care about the production rate. They 
don't want to unload the whole topic to fix the subscription problem.

I've got some questions:

1. How do you handle the race condition when you are trying to unload the 
subscription, and the new consumer wants to subscribe to this subscription at 
the same time? I'm unsure if it has the race condition. I just want to remind 
you about that. :)
2. Would you like to add some restful API design to clarify the implementation?
a. Request method
b. Request path
c. Response code
d. etc.


Thanks for your work.
Mattison
On Jan 11, 2023, 17:01 +0800, Yubiao Feng 
, wrote:
> Hi community
>
> I am starting a DISCUSS for PIP-240: A new API to unload subscriptions.
>
> PIP issue: https://github.com/apache/pulsar/issues/19187
>
> ### Motivation
>
> We sometimes try to unload the topic to resolve some consumption-stop
> issues. But the unloading topic will also impact the producer side.
>
> ### Goal
>
> Providing a new API to unload the subscription dimension triggers
> reconnection of all consumers on that subscription and reconnection is
> guaranteed by the client. The API will be used in these ways:
> - unload special subscription of one topic(or partitioned topic)
> - unload all subscriptions of one topic(or partitioned topic)
> - unload subscriptions of one topic(or partitioned topic) by regular
> expression
> - If a reader's subscription name is not set, a random subscription name
> prefixed with 'multiTopicsReader-' or 'reader-' will be used, and users can
> uninstall these subscriptions using regular expressions.
>
> In addition to triggering consumer disconnection, Unloading Subscribers
> will restart the Dispatcher, which resets the redeliver message queue and
> delayed message queue in the Broker's memory, which can help resolve issues
> caused by an abnormal dispatcher state. However, the execution flow of
> Unloading Subscribers does not include a restart of the Managed Cursor
> related to this dispatcher; if there is a problem with the cursor, we can
> only rely on the unload topic to solve it.
>
> Note: From the client's perspective, this connection may be shared by
> consumers, producers, and transactions, so Unloading Subscribers maybe
> impact the producer and transaction.
>
>  These scenarios are not supported
> - Functions `message-dedup`, `geo-replication,` and `shadow-topic` also
> read messages from the topic, but Unloading subscribers will not support
> triggering restarts of these three functions( because the cursor is used
> directly to read the data in these scenarios, not the consumer or reader ).
> - The Compression task(subscription name is `__compaction`) also use a
> reader to read data, but Unloading Subscribers does not support it because
> this task creates a new reader each time it starts.
> - Do not support all topics related to Transaction features.
> - `__transaction_buffer_snapshot` works with the task TB recover, and
> this task will create a new reader each time they start.
> - `__transaction_pending_ack` works with the task Transaction Pending Ack
> Store replay, and this task will use managed cursor directly to read data.
> - `__transaction_log_xxx` works with the task Transaction Log, which will
> use managed cursor directly to read data.
> - `transaction_coordinator_assign` No data will be written on this topic.
>
>  Special system topic supports
> The system topic `__change_events` is used to support topic-level policies,
> there may also be some message delivery issues in this scenario, so
> Unloading Subscribers will support this topic.
>
> ### API Changes
>
>  For persistent topic
> ```
> pulsar-admin persistent unload {topic_name} -s {sub_name}
> ```
>
>  For non-persistent topic
> ```
> pulsar-admin non-persistent unload {topic_name} -s {sub_name}
> ```
>
>  Explain the param `-s`
> - set param `-s` to special sub name to unload special subscription
> - set param `-s` to `**` to unload all subscriptions under this topic
> - set param `-s` to `regexp` to unload a batch subscriptions under this
> topic
>
>
> Thanks
> Yubiao Feng

Re: [DISCUSS] Introduce oshi library to sensory OS resources

[mattisonchao]

I will draft a PIP for it.

Mattison
On Jan 9, 2023, 22:30 +0800, Dave Fisher , wrote:
> I think oshi is very interesting. I really would like a discussion about how 
> we will use it along with how we can test for any gaps it might have. Testing 
> will need to consider many more types of platforms than laptops and cloud 
> providers. Those of us who use AWS are aware that there may be gaps in 
> features like nic speed. The software does have a test class. One step might 
> be to start trying different VMs in various environments to understand any 
> gaps. Best Regards, Dave

Re: [DISCUSS] Reject create non-existent persistent partitions.

[mattisonchao]

Hi, All

I have another question that needs to discuss.

Should we allow the user to create the non-partitioned topic name like 
`persistent://tenant/namespace/localname-partition-0`?

If so, this is a little confusing with the partitioned topic.

e.g.:
TopicName#isPartitioned method.

Best,
Mattison
On Dec 28, 2022, 12:43 +0800, mattisonc...@gmail.com, wrote:
> Hi, All
>
> I'd like to start a discussion of this behaviour change as follow.
>
> The issue is described here:  https://github.com/apache/pulsar/issues/19085
> And the fix PR here:  https://github.com/apache/pulsar/pull/19086
>
> ---
>
> Behaviour change:
>
> Before: we can create non-existent persistent partitions.
>
> After: we will get `PulsarClientException.TopicDoesNotExistException` when we 
> create non-existent persistent partitions.
>
> Please feel free to leave comments if you have any concerns.
>
> Best,
> Mattison

Re: [DISCUSS] Reject create non-existent persistent partitions.

[mattisonchao]

Hi, Enrico
> Please note that the new test case is about non-persistent topics
I'm sorry, It's my mistake, the non-persistent already fix this problem, I need 
to change the test topic name to persistent.

Best,
Mattison
On Dec 28, 2022, 16:27 +0800, Enrico Olivelli , wrote:
> I agree with you.
>
> Please note that the new test case is about non-persistent topics
>
> is it expected ?
>
> Enrico
>
> Il giorno mer 28 dic 2022 alle ore 07:58 Yubiao Feng
>  ha scritto:
> >
> > Hi qiang
> >
> > I think this is a necessary fix, and it would be nice if more explicit
> > errors were given to the client.
> >
> > Thanks
> > Yubiao
> >
> > On Wed, Dec 28, 2022 at 12:43 PM  wrote:
> >
> > > > Hi, All
> > > >
> > > > I'd like to start a discussion of this behaviour change as follow.
> > > >
> > > > The issue is described here:
> > > > https://github.com/apache/pulsar/issues/19085
> > > > And the fix PR here: https://github.com/apache/pulsar/pull/19086
> > > >
> > > > ---
> > > >
> > > > Behaviour change:
> > > >
> > > > Before: we can create non-existent persistent partitions.
> > > >
> > > > After: we will get `PulsarClientException.TopicDoesNotExistException` 
> > > > when
> > > > we create non-existent persistent partitions.
> > > >
> > > > Please feel free to leave comments if you have any concerns.
> > > >
> > > > Best,
> > > > Mattison
> > > >

[DISCUSS] Reject create non-existent persistent partitions.

[mattisonchao]

Hi, All

I'd like to start a discussion of this behaviour change as follow.

The issue is described here:  https://github.com/apache/pulsar/issues/19085
And the fix PR here:  https://github.com/apache/pulsar/pull/19086

---

Behaviour change:

Before: we can create non-existent persistent partitions.

After: we will get `PulsarClientException.TopicDoesNotExistException` when we 
create non-existent persistent partitions.

Please feel free to leave comments if you have any concerns.

Best,
Mattison

Re: [Vote] PIP-231: Add metric for topic load failed

[mattisonchao]

+1

Best,
Mattison
On Dec 27, 2022, 16:27 +0800, Jiuming Tao , 
wrote:
> Hello pulsar community,
>
> I'm starting the VOTE for PIP-231: Add metric for topic load failed.
>
> Motivation:
> Currently, we have a `topic_load_times` metric to track how long a topic
> load succeeds.
> But when loading a topic, there may be some chances the topic load failed
> due to MetadataStore or Bookkeeper, and we don't have related metrics to
> track it.
>
> For more details, please read the PIP at
> https://github.com/apache/pulsar/issues/18979
> Discuss thread:
> https://lists.apache.org/thread/h02wxj1yclo1o3r7otf9gp38hs0g03ym
>
> Thanks,
> Tao Jiuming

Re: [DISCUSS] Change the default IO threads and listener threads of Java Client

[mattisonchao]

+1

My concern is whether this change will affect some users who are creating many 
clients. I think we can wait for other users to confirm it. (If this will be 
affected, maybe we can give it a max_io_thread_num and then expand the size 
from 1 to max_io_thread_num when adding a new consumer or producer)


Best,
Mattison
On Dec 20, 2022, 11:17 +0800, PengHui Li , wrote:
> Hi all,
>
> I noticed the Java Client (I haven't checked other clients) uses 1 IO
> thread and 1 listener
> thread by default. It will require users to update the thread configuration
> if they have
> multiple cores and desired high throughput.
>
> Here is the example that we change to 16 IO threads in
> openmessaging benchmark
> https://github.com/openmessaging/benchmark/blob/master/driver-pulsar/pulsar.yaml#L22
>
> We can apply the configuration of the threads based on the CPU cores. So
> that for the
> most common cases, users don't need to touch the thread configuration.
>
> ```
> private int numIoThreads = Runtime.getRuntime().availableProcessors();
> private int numListenerThreads = Runtime.getRuntime().availableProcessors();
> ```
>
> WDYT?
>
> Thanks,
> Penghui

Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.

[mattisonchao]

Agree +1.

I remember I mentioned this idea before
https://lists.apache.org/thread/qwcxrr7o29q66dgs4lqkk1nvvvhbjl2s

FYI: 
https://github.com/eclipse-vertx/vert.x/blob/d711e50174f84a06744c855dd29222beaa97bdeb/src/main/java/io/vertx/core/impl/btc/BlockedThreadChecker.java

IMO, we can use one proposal to tidy up the thread pool usage that includes
the following steps:

1. Collect and normalise the use of thread pools.
2. Introduce the block thread checker to check blocking calls. Set default 
enable=false.
3. Introduce the thread auto switcher to avoid sensitive threads(zk thread) 
running unrelated tasks in the CompletableFuture invoke chain.
4. Test and fix all of the blocking behaviour.
5. set default enable=true.


Best
Mattison
On Dec 19, 2022, 20:17 +0800, adobewjl , wrote:
> Hello pulsar community,
> I've opened `PIP-232: Introduce thread monitor to check if thread is blocked 
> for long time.` to discuss.
> For more details, please read the PIP at 
> https://github.com/apache/pulsar/issues/18985
> I'm looking forward to hearing what you think.
> Also the demo PR link at https://github.com/apache/pulsar/pull/18958

[DISCUSS] Introduce oshi library to sensory OS resources

[mattisonchao]

Hi, All

I would like to introduce a new library oshi[1] to help Apache Pulsar sensory 
OS resources. It can help us to get away from the complex file manipulation and 
cross-platform compatibility issues in some operating systems.

code example:   https://github.com/apache/pulsar/pull/18984

Please feel free to left comments.


Best,
Mattison


[1] https://github.com/oshi/oshi

Re: [VOTE] Pulsar Release 2.11.0 Candidate-2

[mattisonchao]

Hi, All

I found two regressions that need to block this release.

https://github.com/apache/pulsar/pull/18816
https://github.com/apache/pulsar/pull/18812


Best,
Mattison
On Dec 8, 2022, 07:15 +0800, Massimiliano Mirelli 
, wrote:
> Hi,
> I am running some long-lasting tests on this rc at the moment. Would it be
> OK to close the vote EOW or could you give me another 20h?
> Thank you,
> Max
>
> On Wed, 7 Dec 2022, 18:47 Nicolò Boschi,  wrote:
>
> > Just realized that the error doesn't block the producer but it's only
> > related to some DNS caching optimizations.
> > I believe we can proceed with the current rc
> >
> > Nicolò Boschi
> >
> >
> > Il giorno mer 7 dic 2022 alle ore 17:40 Nicolò Boschi <
> > boschi1...@gmail.com>
> > ha scritto:
> >
> > > > I'm sorry but I think I've found a blocker issue.
> > > >
> > > > pulsar-perf doesn't support jdk17.
> > > > I know client compatibility targets jdk8 but the image uses jdk17 and
> > this
> > > > may be very annoying if pulsar-perf is not usable within the official
> > > > docker image.
> > > >
> > > > btw I've send out the fix: https://github.com/apache/pulsar/pull/18806
> > > >
> > > >
> > > > Nicolò Boschi
> > > >
> > > >
> > > > Il giorno mer 7 dic 2022 alle ore 16:53 Zixuan Liu 
> > ha
> > > > scritto:
> > > >
> > > > >> Ok, sounds good.
> > > > >>
> > > > >> Thanks,
> > > > >> Zixuan
> > > > >>
> > > > >> PengHui Li  于2022年12月7日周三 16:10写道：
> > > > >>
> > > > > > >> > > I'm wondering whether affecting the resource quota. Could you
> > confirm
> > > > > >> > that?
> > > > > >> >
> > > > > >> > Hmmm, I haven't heard of anyone using this feature.
> > > > > >> > I think it should be ok for a standalone.
> > > > > >> >
> > > > > >> > Penghui
> > > > > >> >
> > > > > >> > On Wed, Dec 7, 2022 at 1:33 PM Zixuan Liu  
> > > > > >> > wrote:
> > > > > >> >
> > > > > > > >> > > > If it only affects standalone. I think it’s ok.
> > > > > > >> > >
> > > > > > >> > > Right.
> > > > > > >> > >
> > > > > > > >> > > > For standalone, multiple bundles help nothing, right?
> > > > > > >> > >
> > > > > > >> > > I'm wondering whether affecting the resource quota. Could you
> > confirm
> > > > > >> > that?
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Zixuan
> > > > > > >> > >
> > > > > > >> > > PengHui Li  于2022年12月7日周三 12:53写道：
> > > > > > >> > >
> > > > > > > >> > > > Hi Zixuan,
> > > > > > > >> > > >
> > > > > > > >> > > > If it only affects standalone. I think it’s ok.
> > > > > > > >> > > > For standalone, multiple bundles help nothing, right?
> > > > > > > >> > > > No rebalance will happen for a standalone.
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Penghui
> > > > > > > >> > > >
> > > > > > > > >> > > > > On Dec 7, 2022, at 11:23, Yunze Xu 
> > > > > > > > >> > > > >  > > >
> > > > > > >> > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > The change of a default value is acceptable in a 
> > > > > > > > >> > > > > major release.
> > > > >> But
> > > > > > > > >> > > > > since it's changed back in the next 2.12 release, it 
> > > > > > > > >> > > > > could be a
> > > > > >> > little
> > > > > > > > >> > > > > confusing. My perspective is to include this PR in 
> > > > > > > > >> > > > > the 2.11.0
> > > > > >> > release.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks,
> > > > > > > > >> > > > > Yunze
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Wed, Dec 7, 2022 at 11:00 AM Zixuan Liu 
> > > > > > > > >> > > > > 
> > > > > >> > wrote:
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Pulsar 2.11 standalone breaks the bundle of 
> > > > > > > > > >> > > > >> namespace policy.
> > I'm
> > > > > > > >> > > > wondering
> > > > > > > > > >> > > > >> whether blocking the Pulsar 2.11 release.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> See https://github.com/apache/pulsar/pull/18755
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Thanks,
> > > > > > > > > >> > > > >> Zixuan
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Enrico Olivelli  
> > > > > > > > > >> > > > >> 于2022年12月6日周二 21:26写道：
> > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >>> +1 (binding)
> > > > > > > > > > >> > > > >>> - verified checksums, digests
> > > > > > > > > > >> > > > >>> - built from sources
> > > > > > > > > > >> > > > >>> - tested the docker image built from sources 
> > > > > > > > > > >> > > > >>> and the docker
> > > > >> image
> > > > > > > > > > >> > > > >>> provided as convenience with the VOTE
> > > > > > > > > > >> > > > >>> - run successfully the Jakarta JMS 2.0 TCK 
> > > > > > > > > > >> > > > >>> against those two
> > > > >> docker
> > > > > > > >> > > > images
> > > > > > > > > > >> > > > >>>
> > > > > > > > > > >> > > > >>> My problem with the JMS TCK is that the 
> > > > > > > > > > >> > > > >>> behaviour of "delete
> > > > >> topic"
> > > > > > > > > > >> > > > >>> with "force=true" changed, and now if the 
> > > > > > > > > > 

Re: [DISCUSS] How to handle broker public API changes

[mattisonchao]

> I'm afraid it's very hard to avoid these API changes. Take theprotocol 
> handler as example, it could make use of nearly all modulesvia the 
> `PulsarService` object. The cost to keep the compatibilitymight be high so 
> that much legacy code could be left. For example,each time a new argument is 
> added to a method, we have to add anoverload to keep the compatibility. It 
> could be confusing to peoplewho don't know these extensions and they might 
> wonder why an overloadis needed that is never used in the Pulsar's main repo.
Yes, I totally agree with your point. it's a trade-off,  I think we can hold 
this discussion until the ecosystem becomes bigger and this problem becomes 
more prominent.

Best,
Mattison

On Dec 7, 2022, 13:37 +0800, Yunze Xu , wrote:
> I'm afraid it's very hard to avoid these API changes. Take the
> protocol handler as example, it could make use of nearly all modules
> via the `PulsarService` object. The cost to keep the compatibility
> might be high so that much legacy code could be left. For example,
> each time a new argument is added to a method, we have to add an
> overload to keep the compatibility. It could be confusing to people
> who don't know these extensions and they might wonder why an overload
> is needed that is never used in the Pulsar's main repo.
>
> The root cause is that we don't have an API abstraction for the
> pulsar-broker module, like the pulsar-client-api module to the
> pulsar-client mode. But it could be a huge project to add something
> like pulsar-broker-api.
>
> Thanks,
> Yunze
>
> On Wed, Dec 7, 2022 at 1:21 PM  wrote:
> >
> > > > It would help me understand if you would explain how apis were changed 
> > > > in this PR
> > Sorry, I explained above. it's a small break, maybe it just breaks some 
> > unit test mock, but it can be used as an example.
> > > > We should be sure to track and then highlight API changes in release 
> > > > documents.
> > > > API changes should be held until the next major version (now 2.12) and 
> > > > be documented with that release.
> > > > I think we need to be explicit in our GitHub PR template. I don’t have 
> > > > a suggested edit yet, but the idea is to add text about
> > > > 1. How the PR changes the API2. Which PIP authorizes it
> > +1 for an awesome idea.
> >
> > Am I just wondering if we can avoid breaking it? maybe we can give the old 
> > method a @Deprecated?
> >
> > Best,
> > Mattison
> > On Dec 7, 2022, 12:56 +0800, Dave Fisher , wrote:
> > > > We should be sure to track and then highlight API changes in release 
> > > > documents. API changes should be held until the next major version (now 
> > > > 2.12) and be documented with that release. I think we need to be 
> > > > explicit in our GitHub PR template. I don’t have a suggested edit yet, 
> > > > but the idea is to add text about 1. How the PR changes the API 2. 
> > > > Which PIP authorizes it

Re: [DISCUSS] Remove restful producer component

[mattisonchao]

Hi Haiting
> Is it better to finish it first, and then remove this from the main repo?
I think we can remove it first because it is not a complete feature. Anyone 
interested in working on it can find the code from the git history and migrate 
it to another repo.

Ps: it looks like the current implementation has some limits, maybe another 
contributor like to use a different way.

Please let me know if you have any concerns or if I am missing something. 
Thanks a lot!

Best
Mattison
On Dec 7, 2022, 12:10 +0800, Haiting Jiang , wrote:
> Hi Mattison
>
> What's the status of "moving this feature to another project"
> Is it better to finish it first, and then remove this from the main repo?
>
> Thanks,
> Haiting
>
> On Tue, Dec 6, 2022 at 6:37 PM  wrote:
> >
> >
> > Hello, everyone.
> >
> > I'd like to start the discussion about `Remove restful producer component`. 
> > The Github repository path is here[1].
> >
> > As discussed before[2], moving this feature to another project is better. 
> > Also, we didn't provide the consumer part in the pulsar repo. I think it's 
> > a good chance to remove it after 2.11 is released.
> >
> > Best,
> > Mattison
> >
> > [1] 
> > https://github.com/apache/pulsar/tree/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest
> > [2] https://lists.apache.org/thread/fl2rbb6sxlzwgkt7ybx4jxfkfnlb27z1
> >
> >

Re: [DISCUSS] How to handle broker public API changes

[mattisonchao]

>  It would help me understand if you would explain how apis were changed in 
> this PR
Sorry, I explained above. it's a small break, maybe it just breaks some unit 
test mock, but it can be used as an example.
> We should be sure to track and then highlight API changes in release 
> documents.
> API changes should be held until the next major version (now 2.12) and be 
> documented with that release.
> I think we need to be explicit in our GitHub PR template. I don’t have a 
> suggested edit yet, but the idea is to add text about
> 1. How the PR changes the API2. Which PIP authorizes it
+1 for an awesome idea.

Am I just wondering if we can avoid breaking it? maybe we can give the old 
method a @Deprecated?

Best,
Mattison
On Dec 7, 2022, 12:56 +0800, Dave Fisher , wrote:
> We should be sure to track and then highlight API changes in release 
> documents. API changes should be held until the next major version (now 2.12) 
> and be documented with that release. I think we need to be explicit in our 
> GitHub PR template. I don’t have a suggested edit yet, but the idea is to add 
> text about 1. How the PR changes the API 2. Which PIP authorizes it

Re: [DISCUSS] How to handle broker public API changes

[mattisonchao]

> Could you please share more details about how the change breaks the broker 
> public API?

For example, this PR[1] change the public API from non-argument to one 
argument. It will cause other libraries also change for it.

I'm not sure that ensuring the compatibility of method signatures is 
worthwhile, but I think it's a bit annoying for other libraries.

https://github.com/apache/pulsar/pull/16425/files#r1041758135

Best,
Mattison
On Dec 7, 2022, 12:50 +0800, PengHui Li , wrote:
> Hi mattison
>
> Could you please share more details about how the change breaks the broker 
> public API?
> It will help us to understand the compatibility issue.
>
> Thanks,
> Penghui
>
> > On Dec 7, 2022, at 12:39, mattisonc...@gmail.com wrote:
> >
> > Hi, All
> >
> > Recently, I realised we don't have any rules for breaking pulsar broker 
> > public API. [1]
> >
> > Since pulsar has extended abilities like protocol handler, additional 
> > servlet, interceptor etc., many libraries rely on broker public API.
> > I'm not sure if we've got to ensure compatibility for this. Otherwise, it 
> > introduces much work in resolving compatibility for other OSS contributors.
> >
> > Best,
> > Mattison
> >
> >
> > [1] https://github.com/apache/pulsar/pull/16425
>

[DISCUSS] How to handle broker public API changes

[mattisonchao]

Hi, All

Recently, I realised we don't have any rules for breaking pulsar broker public 
API. [1]

Since pulsar has extended abilities like protocol handler, additional servlet, 
interceptor etc., many libraries rely on broker public API.
I'm not sure if we've got to ensure compatibility for this. Otherwise, it 
introduces much work in resolving compatibility for other OSS contributors.

Best,
Mattison


[1] https://github.com/apache/pulsar/pull/16425

[DISCUSS] Remove restful producer component

[mattisonchao]

Hello, everyone.

I'd like to start the discussion about `Remove restful producer component`. The 
Github repository path is here[1].

As discussed before[2], moving this feature to another project is better. Also, 
we didn't provide the consumer part in the pulsar repo. I think it's a good 
chance to remove it after 2.11 is released.

Best,
Mattison

[1] 
https://github.com/apache/pulsar/tree/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/rest
[2] https://lists.apache.org/thread/fl2rbb6sxlzwgkt7ybx4jxfkfnlb27z1

Re: [Vote] PIP-223: Add metrics for all Rest Endpoints

[mattisonchao]

+1

Best,
Mattison
On Nov 28, 2022, 16:47 +0800, Jiuming Tao , 
wrote:
> Dear Pulsar Community,
>
> Please review and vote on this PIP.
>
> PIP link: https://github.com/apache/pulsar/issues/18560
>
> Discuss thread:
> https://lists.apache.org/thread/z74vcn0yolzzrcc4ftonm9j3nbk4pzxm
>
> Thanks,
> Tao Jiuming

[DISCUSS] Introduce the blocked thread detector

[mattisonchao]

Hello, everyone.

Since Apache Pulsar aims to be a pure asynchronous-based application, we can 
introduce a blocked thread detector to check all threads.

That can help Apache Pulsar detect unexpected blocking calls that are blocking 
other operations.

WDYT?

Best,
Mattison

Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs

[mattisonchao]

+1(non-binding)

Best,
Mattison
On Dec 5, 2022, 15:09 +0800, Zike Yang , wrote:
> +1(non-binding)
>
> Best,
> Zike Yang
>
> On Mon, Dec 5, 2022 at 2:41 PM Baodi Shi  wrote:
> >
> > +1(non-binding)
> >
> > Thanks,
> > Baodi Shi
> >
> > > > 2022年12月5日 12:51，Yunze Xu  写道：
> > > >
> > > > Hi all,
> > > >
> > > > I'm starting the vote for PIP-224: Introduce TopicMessageId for
> > > > consumer's MessageId related APIs:
> > > > https://github.com/apache/pulsar/issues/18616
> > > >
> > > > Here is the discussion thread:
> > > > https://lists.apache.org/thread/jhqy65cdyxzmmxnfsjm8rv9pbk76noxy
> > > >
> > > > The vote will be open for at least 3 days.
> > > >
> > > > Thanks,
> > > > Yunze
> >