Re: [ANNOUNCE] New Committer: Baodi Shi
Congratulations ! Zike Yang 于2023年1月19日周四 23:18写道： > Congratulations! Baodi > > BR, > Zike Yang > > On Thu, Jan 19, 2023 at 6:11 PM Jiuming Tao > wrote: > > > > Congrats > > > > > 2023年1月18日 21:35，Yunze Xu 写道： > > > > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > > Baodi Shi (https://github.com/shibd) to become a committer and we are > > > pleased to announce that he has accepted. > > > > > > Being a committer enables easier contribution to the project since > > > there is no need to go via the patch submission process. This should > > > enable better productivity. > > > > > > Welcome and congratulations, Baodi Shi! > > > > > > Please join us in congratulating and welcoming Baodi Shi onboard! > > > > > > Thanks, > > > Yunze on behalf of the Pulsar PMC > > >
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congratulations! Thanks, Yubiao Feng On Wed, Jan 18, 2023 at 9:50 PM PengHui Li wrote: > Hi all, > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > (https://github.com/congbobo184) as a member of the PMC and we are > pleased to announce that he has accepted. > > He is very active in the community in the past few years and made a lot of > great contributions > such as transactions and schemas. > > Welcome Bo Cong to the Apache Pulsar PMC. > > Best Regards, > Penghui on behalf of the Pulsar PMC >
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congratulations! Thanks, Yunze On Fri, Jan 20, 2023 at 10:42 AM Dave Fisher wrote: > > Bo Cong! Congratulations! > > Best Regards, > Dave > > Sent from my iPhone > > > On Jan 18, 2023, at 5:50 AM, PengHui Li wrote: > > > > Hi all, > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > > (https://github.com/congbobo184) as a member of the PMC and we are > > pleased to announce that he has accepted. > > > > He is very active in the community in the past few years and made a lot of > > great contributions > > such as transactions and schemas. > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > Best Regards, > > Penghui on behalf of the Pulsar PMC >
Re: [DISCUSS] Code freeze for Pulsar 2.12
On Thu, Jan 19, 2023 at 3:11 PM Christophe Bornet wrote: > It's great that we released Pulsar 2.11. It has taken quite some time to > stabilize the release branch and now we have more than 5 months of awesome > features and commits on the master branch that would benefit a lot to our > users. That's why I'd like to propose to start a code freeze for the > release of Pulsar 2.12 with a target release date by mid/end-february. 2.11 has just been released and the release cycle is set to 3 months. It's not a good idea to call for code freezes without appropriate warning, and code freezes don't make sense in general: the "master" branch must always be open for business. There was already a proposal to codify a formal release plan with prescribed timings for release RCs and stabilization. (https://github.com/apache/pulsar/issues/15966) It was actually discussed already very recently ((offline) that it's a good time to formally adopt it for the next release.
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Bo Cong! Congratulations! Best Regards, Dave Sent from my iPhone > On Jan 18, 2023, at 5:50 AM, PengHui Li wrote: > > Hi all, > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > (https://github.com/congbobo184) as a member of the PMC and we are > pleased to announce that he has accepted. > > He is very active in the community in the past few years and made a lot of > great contributions > such as transactions and schemas. > > Welcome Bo Cong to the Apache Pulsar PMC. > > Best Regards, > Penghui on behalf of the Pulsar PMC
Re: [DISCUSS] Code freeze for Pulsar 2.12
Christophe, Given the Chinese New Year what freeze date is being proposed? Best, Dave Sent from my iPhone > On Jan 19, 2023, at 6:31 PM, Yunze Xu wrote: > > In addition, next week is the Chinese New Year [1] in China and there > is a long holiday (a week) for Chinese developers. I hope we can delay > this release for a while. > > [1] https://en.wikipedia.org/wiki/Chinese_New_Year > > Thanks, > Yunze > >> On Fri, Jan 20, 2023 at 10:23 AM Yunze Xu wrote: >> >> I would like to include PIP-224 (even and PIP-229) in the next major >> releases. These two PIPs have some impacts on the API and could bring >> many benefits to ecosystem developers. But unfortunately the first PR >> of PIP-224 [1] is still not reviewed by anyone. The code has already >> been added locally and only requires some rebase to resolve conflicts. >> >> [1] https://github.com/apache/pulsar/pull/19158 >> >> Thanks, >> Yunze >> >>> On Fri, Jan 20, 2023 at 7:46 AM wrote: >>> >>> Isn't the next version LTS 3.0? >>> >>> Best >>> Mattison >>> On Jan 20, 2023, 07:11 +0800, Christophe Bornet , >>> wrote: Hi Pulsar community, It's great that we released Pulsar 2.11. It has taken quite some time to stabilize the release branch and now we have more than 5 months of awesome features and commits on the master branch that would benefit a lot to our users. That's why I'd like to propose to start a code freeze for the release of Pulsar 2.12 with a target release date by mid/end-february. Hopefully this release will be easier to stabilize but we don't know for sure, so better to start the release activities asap. We also need a release manager. Nicolo proposed himself last time but had to hand over because of his holiday schedule. So Nicolo, maybe you'd like to propose yourself again for this one ? Otherwise I'm happy to volunteer. Let me know what you think. Cheers Christophe
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congratulations! Bo Cong On Fri, Jan 20, 2023 at 7:17 AM guo jiwei wrote: > Congratulations! Bo Cong > > Regards > Jiwei Guo (Tboy) > > On Fri, Jan 20, 2023 at 6:36 AM Neng Lu wrote: > > > > Congratulations! > > > > On Thu, Jan 19, 2023 at 7:18 AM Zike Yang wrote: > > > > > Congratulations! Bo Cong > > > > > > BR, > > > Zike Yang > > > > > > On Thu, Jan 19, 2023 at 6:11 PM Jiuming Tao > > > wrote: > > > > > > > > Congrats！ > > > > > > > > > 2023年1月18日 21:50，PengHui Li 写道： > > > > > > > > > > Hi all, > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo > > > Cong > > > > > (https://github.com/congbobo184) as a member of the PMC and we are > > > > > pleased to announce that he has accepted. > > > > > > > > > > He is very active in the community in the past few years and made a > > > lot of > > > > > great contributions > > > > > such as transactions and schemas. > > > > > > > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > > > > > > > Best Regards, > > > > > Penghui on behalf of the Pulsar PMC > > > > > > > > -- Regards Nitin Goyal
Re: [DISCUSS] Code freeze for Pulsar 2.12
In addition, next week is the Chinese New Year [1] in China and there is a long holiday (a week) for Chinese developers. I hope we can delay this release for a while. [1] https://en.wikipedia.org/wiki/Chinese_New_Year Thanks, Yunze On Fri, Jan 20, 2023 at 10:23 AM Yunze Xu wrote: > > I would like to include PIP-224 (even and PIP-229) in the next major > releases. These two PIPs have some impacts on the API and could bring > many benefits to ecosystem developers. But unfortunately the first PR > of PIP-224 [1] is still not reviewed by anyone. The code has already > been added locally and only requires some rebase to resolve conflicts. > > [1] https://github.com/apache/pulsar/pull/19158 > > Thanks, > Yunze > > On Fri, Jan 20, 2023 at 7:46 AM wrote: > > > > Isn't the next version LTS 3.0? > > > > Best > > Mattison > > On Jan 20, 2023, 07:11 +0800, Christophe Bornet , > > wrote: > > > Hi Pulsar community, > > > > > > It's great that we released Pulsar 2.11. It has taken quite some time to > > > stabilize the release branch and now we have more than 5 months of awesome > > > features and commits on the master branch that would benefit a lot to our > > > users. That's why I'd like to propose to start a code freeze for the > > > release of Pulsar 2.12 with a target release date by mid/end-february. > > > Hopefully this release will be easier to stabilize but we don't know for > > > sure, so better to start the release activities asap. > > > We also need a release manager. Nicolo proposed himself last time but had > > > to hand over because of his holiday schedule. So Nicolo, maybe you'd like > > > to propose yourself again for this one ? Otherwise I'm happy to volunteer. > > > > > > Let me know what you think. > > > > > > Cheers > > > > > > Christophe
Re: [VOTE] PIP-224: Introduce TopicMessageId for consumer's MessageId related APIs
Hi guys,
Please help review https://github.com/apache/pulsar/pull/19158, which
has been pending for a while. I see 2.12.0 (or 3.0.0?) is coming, I
hope we can include this PIP in the next major release.
Thanks,
Yunze
On Wed, Dec 28, 2022 at 10:40 PM Yunze Xu wrote:
>
> Hi Enrico,
>
> Yeah, I've discussed it privately with Hang and he had the same
> concern with you and. So I changed the `Schema` to a
> more simple class:
>
> ```java
>
> class TopicMessageIdSerDes {
> public static byte[] serialize(TopicMessageId topicMessageId) {/* ... */}
> public static TopicMessageId deserialize(byte[] bytes) {/* ... */}
> }
> ```
>
> You can see it and the updated Alternatives section in
> https://github.com/apache/pulsar/issues/18616
>
> Thanks,
> Yunze
>
> On Wed, Dec 28, 2022 at 4:22 PM Enrico Olivelli wrote:
> >
> > Schema doesn't make much sense to me.
> >
> > Schema is a Pulsar Client API to deal with Message serialisation, it
> > is not a general purpose serde framework.
> >
> > I still approve the PIP overall but I don't think we should expose
> > such things to the user facing APIs
> >
> > Il giorno mer 28 dic 2022 alle ore 09:15 Hang Chen
> > ha scritto:
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Hang
> > >
> > > Yunze Xu 于2022年12月28日周三 11:53写道：
> > > >
> > > > After discussing with @hangc0276 offline, I decided to add a
> > > > `Schema` implementation in this proposal to serialize
> > > > both the owner topic and the base `MessageId`. You can see the latest
> > > > update on GitHub. If any of you have any concern, feel free to let me
> > > > know.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Dec 26, 2022 at 1:22 AM Enrico Olivelli
> > > > wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il Ven 23 Dic 2022, 05:46 Yunze Xu ha
> > > > > scritto:
> > > > >
> > > > > > It needs the 3rd binding +1 yet. Could anyone else take a look?
> > > > > >
> > > > > > Thanks,
> > > > > > Yunze
> > > > > >
> > > > > > On Wed, Dec 21, 2022 at 3:21 PM 丛搏 wrote:
> > > > > > >
> > > > > > > Hi Yunze,
> > > > > > >
> > > > > > > add `TopicMessageId ` will couple messageID and `topic name`
> > > > > > > together,
> > > > > > > which is very unclear for non-partition-topic.
> > > > > > >
> > > > > > > ```
> > > > > > > void seek(String topicName, MessageId messageId) throws
> > > > > > PulsarClientException;
> > > > > > > List> getLastTopicMessageId() throws
> > > > > > > PulsarClientException;
> > > > > > > ```
> > > > > > > If the interface is designed in this way, it may be simpler,
> > > > > > > easier to
> > > > > > > understand, and more intuitive for users, and MessageID will not
> > > > > > > be
> > > > > > > coupled with TopicName.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Bo
> > > > > > >
> > > > > > > Yunze Xu 于2022年12月16日周五 15:31写道：
> > > > > > > >
> > > > > > > > Yeah, it's an implementation detail and I will keep the same
> > > > > > > > semantics
> > > > > > > > with the latest master when I push my PR.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yunze
> > > > > > > >
> > > > > > > > On Fri, Dec 16, 2022 at 3:03 PM 丛搏 wrote:
> > > > > > > > >
> > > > > > > > > if you don't change this in PIP-229 or PIP-224, I will create
> > > > > > > > > a new
> > > > > > > > > PIP to handle the `BatchMessageIdImpl` and `MessageIdImpl`
> > > > > > > > > `compareTo()` method, now I have no problem with this PIP
> > > > > > > > > +1 (non-binding)
> > > > > > > > > Sorry to bother this PIP vote.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Bo
> > > > > > > > >
> > > > > > > > > Yunze Xu 于2022年12月16日周五
> > > > > > > > > 11:58写道：
> > > > > > > > > >
> > > > > > > > > > If this breaking change can pass the PMC votes, I will keep
> > > > > > > > > > the new
> > > > > > > > > > semantics in PIP-229. Otherwise, it would not make sense to
> > > > > > > > > > adopt
> > > > > > the
> > > > > > > > > > new semantics in PIP-229.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Yunze
> > > > > > > > > >
> > > > > > > > > > On Fri, Dec 16, 2022 at 11:46 AM Yunze Xu
> > > > > > > > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > I cannot find any confusing code from the proposal
> > > > > > > > > > > itself. Could
> > > > > > you
> > > > > > > > > > > point it out? If you are mentioning the `legacyCompare`
> > > > > > > > > > > and
> > > > > > `compare`
> > > > > > > > > > > methods in #18890 [1], it's not related to this proposal.
> > > > > > > > > > > And I
> > > > > > have
> > > > > > > > > > > opened PIP-229 [2] for discussion.
> > > > > > > > > > >
> > > > > > > > > > > BTW, the PIP-229 itself doesn't mention the compare
> > > > > > > > > > > logic. But
> > > > > > I'm not
> > > > > > > > > > > going to adopt the new semantics because it's actually a
> > > > > > > > > > > breaking
> > > > > > > > > > > change, just as I've replied. You might think it's a bug,
> > > > > > >
Re: [DISCUSS] Code freeze for Pulsar 2.12
I would like to include PIP-224 (even and PIP-229) in the next major releases. These two PIPs have some impacts on the API and could bring many benefits to ecosystem developers. But unfortunately the first PR of PIP-224 [1] is still not reviewed by anyone. The code has already been added locally and only requires some rebase to resolve conflicts. [1] https://github.com/apache/pulsar/pull/19158 Thanks, Yunze On Fri, Jan 20, 2023 at 7:46 AM wrote: > > Isn't the next version LTS 3.0? > > Best > Mattison > On Jan 20, 2023, 07:11 +0800, Christophe Bornet , > wrote: > > Hi Pulsar community, > > > > It's great that we released Pulsar 2.11. It has taken quite some time to > > stabilize the release branch and now we have more than 5 months of awesome > > features and commits on the master branch that would benefit a lot to our > > users. That's why I'd like to propose to start a code freeze for the > > release of Pulsar 2.12 with a target release date by mid/end-february. > > Hopefully this release will be easier to stabilize but we don't know for > > sure, so better to start the release activities asap. > > We also need a release manager. Nicolo proposed himself last time but had > > to hand over because of his holiday schedule. So Nicolo, maybe you'd like > > to propose yourself again for this one ? Otherwise I'm happy to volunteer. > > > > Let me know what you think. > > > > Cheers > > > > Christophe
[VOTE] Pulsar Client C++ Release 3.1.1 Candidate 1
This is the first release candidate for Apache Pulsar Client C++, version 3.1.1. It fixes the following issues: https://github.com/apache/pulsar-client-cpp/issues?q=label%3Arelease%2F3.1.1 *** Please download, test and vote on this release. This vote will stay open for at least 72 hours *** Note that we are voting upon the source (tag), binaries are provided for convenience. Source and binary files: https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-cpp/pulsar-client-cpp-3.1.1-candidate-1/ SHA-512 checksums: ff7a98e8702d1086150cc1cbd10827b78d2d23cd7ca639d5590fb37d122b68c7ed70c2e38a24c94375871d868f9f04263b8991668e8498664bc3c1fa644b0be0 apache-pulsar-client-cpp-3.1.1.tar.gz The tag to be voted upon: v3.1.1-candidate-1 (24887d9f43471d6b333179000f9c931d64832037) https://github.com/apache/pulsar-client-cpp/releases/tag/v3.1.1-candidate-1 Pulsar's KEYS file containing PGP keys you use to sign the release: https://dist.apache.org/repos/dist/dev/pulsar/KEYS Please download the source package, and follow the README to compile and test. -- Matteo Merli
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congratulations! Bo Cong Regards Jiwei Guo (Tboy) On Fri, Jan 20, 2023 at 6:36 AM Neng Lu wrote: > > Congratulations! > > On Thu, Jan 19, 2023 at 7:18 AM Zike Yang wrote: > > > Congratulations! Bo Cong > > > > BR, > > Zike Yang > > > > On Thu, Jan 19, 2023 at 6:11 PM Jiuming Tao > > wrote: > > > > > > Congrats！ > > > > > > > 2023年1月18日 21:50，PengHui Li 写道： > > > > > > > > Hi all, > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo > > Cong > > > > (https://github.com/congbobo184) as a member of the PMC and we are > > > > pleased to announce that he has accepted. > > > > > > > > He is very active in the community in the past few years and made a > > lot of > > > > great contributions > > > > such as transactions and schemas. > > > > > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > > > > > Best Regards, > > > > Penghui on behalf of the Pulsar PMC > > > > >
Re: [DISCUSS] Code freeze for Pulsar 2.12
Isn't the next version LTS 3.0? Best Mattison On Jan 20, 2023, 07:11 +0800, Christophe Bornet , wrote: > Hi Pulsar community, > > It's great that we released Pulsar 2.11. It has taken quite some time to > stabilize the release branch and now we have more than 5 months of awesome > features and commits on the master branch that would benefit a lot to our > users. That's why I'd like to propose to start a code freeze for the > release of Pulsar 2.12 with a target release date by mid/end-february. > Hopefully this release will be easier to stabilize but we don't know for > sure, so better to start the release activities asap. > We also need a release manager. Nicolo proposed himself last time but had > to hand over because of his holiday schedule. So Nicolo, maybe you'd like > to propose yourself again for this one ? Otherwise I'm happy to volunteer. > > Let me know what you think. > > Cheers > > Christophe
[DISCUSS] Code freeze for Pulsar 2.12
Hi Pulsar community, It's great that we released Pulsar 2.11. It has taken quite some time to stabilize the release branch and now we have more than 5 months of awesome features and commits on the master branch that would benefit a lot to our users. That's why I'd like to propose to start a code freeze for the release of Pulsar 2.12 with a target release date by mid/end-february. Hopefully this release will be easier to stabilize but we don't know for sure, so better to start the release activities asap. We also need a release manager. Nicolo proposed himself last time but had to hand over because of his holiday schedule. So Nicolo, maybe you'd like to propose yourself again for this one ? Otherwise I'm happy to volunteer. Let me know what you think. Cheers Christophe
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congratulations! On Thu, Jan 19, 2023 at 7:18 AM Zike Yang wrote: > Congratulations! Bo Cong > > BR, > Zike Yang > > On Thu, Jan 19, 2023 at 6:11 PM Jiuming Tao > wrote: > > > > Congrats！ > > > > > 2023年1月18日 21:50，PengHui Li 写道： > > > > > > Hi all, > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo > Cong > > > (https://github.com/congbobo184) as a member of the PMC and we are > > > pleased to announce that he has accepted. > > > > > > He is very active in the community in the past few years and made a > lot of > > > great contributions > > > such as transactions and schemas. > > > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > > > Best Regards, > > > Penghui on behalf of the Pulsar PMC > > >
Pulsar Cpp client 3.1.1 release
There is a serious regression bug in Pulsar C++ client 3.1.0 where the client lib is not passing the "authoritative" flag after a lookup redirect. This causes the client to end up in an infinite loop of redirects. The problem was already fixed in https://github.com/apache/pulsar-client-cpp/pull/146, but it still needs to be released. This problem is serious enough that I believe we should get a patch out ASAP. I'll start the work on a 3.1.1 patch release. Matteo -- Matteo Merli
Re: [DISCUSS] PIP-241: TopicEventListener / topic events for the BrokerService
Hi Haiting,
Currently there are no plans to use TopicEvent/EventStage outside of the
TopicEventsListener.
We can refactor that if the situation changes and if these will be reusable
(which I doubt at least for the TopicEvent).
I'd keep it contained in one file/namespace.
On Thu, Jan 19, 2023 at 2:37 AM Haiting Jiang
wrote:
> Hi Andrey,
>
> This PIP makes sense to me.
> A small question: Is it better to define `TopicEvent` and `EventStage`
> outside of `TopicEventsListener`?
>
> Thanks,
> Haiting
>
>
> On Tue, Jan 17, 2023 at 3:51 PM Enrico Olivelli
> wrote:
> >
> > I support this PIP.
> > This will allow Protocol Handlers to have better knowledge of what's
> > happening on the broker.
> >
> > I hope that someday we will be able to refactor Broker internals using
> > this feature and decouple the components.
> >
> > Enrico
> >
> > Il giorno lun 16 gen 2023 alle ore 10:27 Yunze Xu
> > ha scritto:
> > >
> > > +1 to me now.
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Sat, Jan 14, 2023 at 1:29 AM Andrey Yegorov
> > > wrote:
> > > >
> > > > My bad, I pasted the interface code from a branch with experiment to
> cancel
> > > > events. This is no longer needed.
> > > > EventProcessingResult result is irrelevant, I updated the PIP.
> > > >
> > > > It is not in the PR.
> > > >
> > > > On Fri, Jan 13, 2023 at 4:08 AM Yunze Xu
>
> > > > wrote:
> > > >
> > > > > I have some questions about the EventProcessingResult:
> > > > > 1. What's the difference between FAILURE and NOT_ALLOWED?
> > > > > 2. If we need to return the `message`, which indicates the error
> IIUC,
> > > > > would it be better to replace the returned value with a checked
> > > > > exception?
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Fri, Jan 13, 2023 at 12:36 PM Andrey Yegorov <
> ayego...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am starting discussion for PIP-241: TopicEventListener / topic
> events
> > > > > for
> > > > > > the BrokerService.
> > > > > >
> > > > > > PIP issue: https://github.com/apache/pulsar/issues/19224
> > > > > >
> > > > > > ### Motivation
> > > > > >
> > > > > > Some Protocol Handlers may need to know about the topic-specific
> events
> > > > > to
> > > > > > update internal caches and/or state.
> > > > > >
> > > > > > These mechanisms will be useful also for core Pulsar components
> (like the
> > > > > > Transactions subsystem) and probably we would be able to
> simplify the
> > > > > > interaction between the internal components in the broker by
> using an
> > > > > > unified mechanism to handle the lifecycle of topics.
> > > > > >
> > > > > > Specific use cases:
> > > > > >
> > > > > > KOP keeps some state for the topic and needs to handle such
> cases as:
> > > > > >
> > > > > > - Topic Unloaded: release resources dedicated to the topic
> > > > > > - Topic Loaded: trigger loading of components tied to the
> partition
> > > > > > (GroupCoordinator, TransactionManager)
> > > > > > - Topic Deleted: remove any persistent state associated to the
> topic that
> > > > > > is stored in additional side system topics
> > > > > > - Topic Created: the same as “deleted” (ensure that there is no
> state on
> > > > > > system topics related to the new topic)
> > > > > >
> > > > > >
> > > > > > ### Goal
> > > > > >
> > > > > > This PIP defines a set of events needed for the protocol
> handlers (and
> > > > > for
> > > > > > internal broker components) to get notifications about
> topic-specific
> > > > > > events as seen by BrokerService. PIP outlines changes needed for
> protocol
> > > > > > handlers to keep/cache state consistent with BrokerService’s.
> > > > > >
> > > > > > The changes should not affect Pulsar running without protocol
> handlers or
> > > > > > with protocol handlers that do not rely on the new events.
> > > > > >
> > > > > >
> > > > > > ### API Changes
> > > > > >
> > > > > > ```java
> > > > > > /**
> > > > > > * Listener for the Topic events.
> > > > > > */
> > > > > > @InterfaceAudience.LimitedPrivate
> > > > > > @InterfaceStability.Evolving
> > > > > > public interface TopicEventsListener {
> > > > > >
> > > > > > /**
> > > > > > * Types of events currently supported.
> > > > > > * create/load/unload/delete
> > > > > > */
> > > > > > enum TopicEvent {
> > > > > > // create events included into load events
> > > > > > CREATE,
> > > > > > LOAD,
> > > > > > UNLOAD,
> > > > > > DELETE,
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > > * Stages of events currently supported.
> > > > > > * before starting the event/successful completion/failed
> completion
> > > > > > */
> > > > > > enum EventStage {
> > > > > > BEFORE,
> > > > > > SUCCESS,
> > > > > > FAILURE
> > > > > > }
> > > > > >
> > > > > > /**
> > > > > > * Outcome of the listener.
> > > > > > * Ignored for events at final stages (SUCCESS/FAILURE),
> >
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congratulations! Bo Cong BR, Zike Yang On Thu, Jan 19, 2023 at 6:11 PM Jiuming Tao wrote: > > Congrats！ > > > 2023年1月18日 21:50，PengHui Li 写道： > > > > Hi all, > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > > (https://github.com/congbobo184) as a member of the PMC and we are > > pleased to announce that he has accepted. > > > > He is very active in the community in the past few years and made a lot of > > great contributions > > such as transactions and schemas. > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > Best Regards, > > Penghui on behalf of the Pulsar PMC >
Re: [ANNOUNCE] New Committer: Baodi Shi
Congratulations! Baodi BR, Zike Yang On Thu, Jan 19, 2023 at 6:11 PM Jiuming Tao wrote: > > Congrats > > > 2023年1月18日 21:35，Yunze Xu 写道： > > > > The Project Management Committee (PMC) for Apache Pulsar has invited > > Baodi Shi (https://github.com/shibd) to become a committer and we are > > pleased to announce that he has accepted. > > > > Being a committer enables easier contribution to the project since > > there is no need to go via the patch submission process. This should > > enable better productivity. > > > > Welcome and congratulations, Baodi Shi! > > > > Please join us in congratulating and welcoming Baodi Shi onboard! > > > > Thanks, > > Yunze on behalf of the Pulsar PMC >
Re: [DISCUSS] PIP-232: Introduce thread monitor to check if thread is blocked for long time.
I support the motivation and goal. Thanks for driving this feature. Suggestion: 1. Split this PIP into parts, and start with low overhead metrics, like Heesung mentions. 2. Do some solid performance tests to assess the impact of this monitor. I am not sure if we would have performance reduction even if this feature is turned off. If so, we also need to know how much the impact is. Thanks, Haiting On Thu, Jan 12, 2023 at 8:36 PM Lin Lin wrote: > > > I have the following suggestions: > 1. This configuration item should be dynamically updated in the Pulsar > process, only as a means of troubleshooting when problems occur > 2. This configuration item should be turned off by default to avoid impact on > performance
Re: [DISCUSS] Add unified newTableView method in PulsarClient
+1 Haiting On Mon, Jan 16, 2023 at 3:14 PM Enrico Olivelli wrote: > > +1 > > Enrico > > Il Lun 16 Gen 2023, 03:11 PengHui Li ha scritto: > > > +1 > > > > And It might be a typo when adding the table view APIs > > From the example: > > > > > > https://github.com/apache/pulsar/blob/246c2701e5c43e02e9783c82d4d107d06b019951/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/PulsarClient.java#L232-L238 > > > > And from the proposal, the method name is newTableView() > > > > https://github.com/apache/pulsar/issues/12356 > > > > Thanks, > > Penghui > > > > On Sun, Jan 15, 2023 at 6:31 PM Ruguo Yu wrote: > > > > > Hi Community, > > > > > > Currently, we can use PulsarClient to create `Producer`, `Consumer`, > > > `Reader` and `TableView`, releated method as below: > > > > > > ``` > > > > > > ProducerBuilder newProducer(); > > > > > > ProducerBuilder newProducer(Schema schema); > > > > > > ConsumerBuilder newConsumer(); > > > > > > ConsumerBuilder newConsumer(Schema schema); > > > > > > ReaderBuilder newReader(); > > > > > > ReaderBuilder newReader(Schema schema); > > > > > > TableViewBuilder newTableViewBuilder(Schema schema); > > > > > > ``` > > > > > > However, it is obvious that the method of creating `TableView` is not > > > consistent with other methods, and no method with default scheme is > > > provided. > > > > > > > > > > > > ### Motivation > > > > > > Add unified `newTableView(Schema)` method and replace > > > `newTableViewBuilder(Schema)`(attach `@Deprecated` annotation to it) in > > > PulsarClient, which could consistent with `newProducer(Schema)`, > > > `newConsumer(Schema)`, `newReader(Schema)`. > > > > > > In addition, we will provide `newTableView()` method which has default > > > schema. > > > > > > > > > > > > ### Modifications > > > > > > 1. Add `newTableView(Schema)` method and replace > > > `newTableViewBuilder(Schema)`(attach `@Deprecated` annotation to it) in > > > PulsarClient > > > > > > 2. Add `newTableView()` method and Scheme default is `Schema.BYTES` in > > > PulsarClient > > > > > > > > > > > > Thanks > > > > > > Ruguo Yu > > > > > > > >
Re: [DISCUSS] PIP-235: Add metric for subscription back size
Hi Xiao, > New metric pulsar_subscription_back_log_size The metric name is better as `pulsar_subscription_backlog_size`, to keep consistent with `pulsar_storage_backlog_size`. > Since 0 is a valid value for subscription backlog size, change > subscriptionStats.backlogSize to -1 from 0 if request the stats of topic with > subscriptionBacklogSize=false. I am not sure why we need -1 for the case that this feature is turned off. Does the metric just don't appear in the Prometheus result? Thanks, Haiting On Sun, Jan 15, 2023 at 6:44 PM Jiuming Tao wrote: > > Hi Xiao, > > Seems `backlogSize` already existed in the `SubscriptionStatsImpl`, but > didn’t expose in Prometheus, > > We just need to expose it in Prometheus. > > It should exposed in Prometheus when the configuration named > `exposeTopicLevelMetricsInPrometheus` and > `exposeSubscriptionBacklogSizeInPrometheus` is enabled. > > +1 > > Thanks, > Tao Jiuming > > > 2022年12月30日 23:25，萧 易客 写道： > > > > can >
Re: [DISCUSS] PIP-241: TopicEventListener / topic events for the BrokerService
Hi Andrey,
This PIP makes sense to me.
A small question: Is it better to define `TopicEvent` and `EventStage`
outside of `TopicEventsListener`?
Thanks,
Haiting
On Tue, Jan 17, 2023 at 3:51 PM Enrico Olivelli wrote:
>
> I support this PIP.
> This will allow Protocol Handlers to have better knowledge of what's
> happening on the broker.
>
> I hope that someday we will be able to refactor Broker internals using
> this feature and decouple the components.
>
> Enrico
>
> Il giorno lun 16 gen 2023 alle ore 10:27 Yunze Xu
> ha scritto:
> >
> > +1 to me now.
> >
> > Thanks,
> > Yunze
> >
> > On Sat, Jan 14, 2023 at 1:29 AM Andrey Yegorov
> > wrote:
> > >
> > > My bad, I pasted the interface code from a branch with experiment to
> > > cancel
> > > events. This is no longer needed.
> > > EventProcessingResult result is irrelevant, I updated the PIP.
> > >
> > > It is not in the PR.
> > >
> > > On Fri, Jan 13, 2023 at 4:08 AM Yunze Xu
> > > wrote:
> > >
> > > > I have some questions about the EventProcessingResult:
> > > > 1. What's the difference between FAILURE and NOT_ALLOWED?
> > > > 2. If we need to return the `message`, which indicates the error IIUC,
> > > > would it be better to replace the returned value with a checked
> > > > exception?
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Fri, Jan 13, 2023 at 12:36 PM Andrey Yegorov
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I am starting discussion for PIP-241: TopicEventListener / topic
> > > > > events
> > > > for
> > > > > the BrokerService.
> > > > >
> > > > > PIP issue: https://github.com/apache/pulsar/issues/19224
> > > > >
> > > > > ### Motivation
> > > > >
> > > > > Some Protocol Handlers may need to know about the topic-specific
> > > > > events
> > > > to
> > > > > update internal caches and/or state.
> > > > >
> > > > > These mechanisms will be useful also for core Pulsar components (like
> > > > > the
> > > > > Transactions subsystem) and probably we would be able to simplify the
> > > > > interaction between the internal components in the broker by using an
> > > > > unified mechanism to handle the lifecycle of topics.
> > > > >
> > > > > Specific use cases:
> > > > >
> > > > > KOP keeps some state for the topic and needs to handle such cases as:
> > > > >
> > > > > - Topic Unloaded: release resources dedicated to the topic
> > > > > - Topic Loaded: trigger loading of components tied to the partition
> > > > > (GroupCoordinator, TransactionManager)
> > > > > - Topic Deleted: remove any persistent state associated to the topic
> > > > > that
> > > > > is stored in additional side system topics
> > > > > - Topic Created: the same as “deleted” (ensure that there is no state
> > > > > on
> > > > > system topics related to the new topic)
> > > > >
> > > > >
> > > > > ### Goal
> > > > >
> > > > > This PIP defines a set of events needed for the protocol handlers (and
> > > > for
> > > > > internal broker components) to get notifications about topic-specific
> > > > > events as seen by BrokerService. PIP outlines changes needed for
> > > > > protocol
> > > > > handlers to keep/cache state consistent with BrokerService’s.
> > > > >
> > > > > The changes should not affect Pulsar running without protocol
> > > > > handlers or
> > > > > with protocol handlers that do not rely on the new events.
> > > > >
> > > > >
> > > > > ### API Changes
> > > > >
> > > > > ```java
> > > > > /**
> > > > > * Listener for the Topic events.
> > > > > */
> > > > > @InterfaceAudience.LimitedPrivate
> > > > > @InterfaceStability.Evolving
> > > > > public interface TopicEventsListener {
> > > > >
> > > > > /**
> > > > > * Types of events currently supported.
> > > > > * create/load/unload/delete
> > > > > */
> > > > > enum TopicEvent {
> > > > > // create events included into load events
> > > > > CREATE,
> > > > > LOAD,
> > > > > UNLOAD,
> > > > > DELETE,
> > > > > }
> > > > >
> > > > > /**
> > > > > * Stages of events currently supported.
> > > > > * before starting the event/successful completion/failed
> > > > > completion
> > > > > */
> > > > > enum EventStage {
> > > > > BEFORE,
> > > > > SUCCESS,
> > > > > FAILURE
> > > > > }
> > > > >
> > > > > /**
> > > > > * Outcome of the listener.
> > > > > * Ignored for events at final stages (SUCCESS/FAILURE),
> > > > > *
> > > > > */
> > > > > enum EventProcessingOutcome {
> > > > > OK,
> > > > > FAILURE,
> > > > > NOT_ALLOWED
> > > > > }
> > > > >
> > > > > /**
> > > > > * POJO for event processing result (outcome, message)
> > > > > */
> > > > > @ToString(includeFieldNames=true)
> > > > > @Data(staticConstructor="of")
> > > > > class EventProcessingResult {
> > > > > private final EventProcessingOutcome outcome;
> > > > > private final String message;
> > > > > }
> >
Re: [DISCUSS] PIP-239: Retry Mechanism per Message
Hi Nitin,
Michael's point about the decoupling of producer and consumer makes sense.
Your use case is more like a queue model, not a pub-sub model. So the
decoupling is not the issue for you.
But Pulsar is mainly a pub-sub system by nature. So we must take it
into more consideration.
As in your case, If we only provide setting max-retry-time in
`Consumer#reconsumeLater`, and forbid Producer to set this value (as
in Producer API).
Will it solve the problem?
Best,
Haiting
On Tue, Jan 17, 2023 at 3:43 PM Asaf Mesika wrote:
>
> From my knowledge, when a client does negative ack, after a certain amount
> of time, it sends the following command to the broker containing the
> negatively acked messages ids:
>
> message CommandRedeliverUnacknowledgedMessages {
> required uint64 consumer_id = 1;
> repeated MessageIdData message_ids = 2;
> optional uint64 consumer_epoch = 3;
> }
>
>
> On Tue, Jan 17, 2023 at 5:24 AM Nitin Goyal
> wrote:
>
> > Hello all,
> >
> > I have a few questions.
> >
> > 1. When a message is NACKed does it still send the commands to the broker?
> > 2. Do we have any plan to replace the ReconsumeLater function with NACK
> > Backoff interface in future?
> >
> >
> > On Tue, Jan 10, 2023 at 7:17 PM Nitin Goyal
> > wrote:
> >
> > > Hello Team,
> > >
> > > Thanks for pointing these things out.
> > >
> > > First thing is to consider that this addon feature is for the message
> > > which needs to go to DLQ after a retryLater reaches the maximum limit of
> > > consumer level retries. It allows an additional maximum limit if a
> > message
> > > has one at the time of producing.
> > >
> > > There are two types of retry in current architecture which consumers can
> > > go with.
> > >
> > > 1. Process later with an NACK backoff policy: In this case if someone
> > > wants to retry and wants to set a maximum retry at message processor
> > level.
> > > They can get the number of times the message has been retried so far
> > > by calling the `reconsumeTimes` function. they will match the number with
> > > their custom retry. Now there are two things that will happen. Either
> > they
> > > will call ACK that particular message or they have to call the
> > > `reconsumeLater` function based on system requirement. If they call ACK
> > > then the message is marked as delete in the ledger. And other cases would
> > > be explained in point 2.
> > >
> > > 2. By calling reconsumeLater: In some real cases if someone is calling
> > > reconsumeLater with a delay like they want to process a message again but
> > > after a particular time. so the new message gets created in the consumer
> > > RETRY topic which was again processed by the consumer (they may call the
> > > reconsumeLater again if the system requires based on use cases.). Now if
> > > the message reaches its retry limit which was configured at consumer
> > level
> > > they will be sent to the DLQ message.
> > >
> > > So my proposal will not make any changes to 1st scenario which is to use
> > > NACK backoff for retrying the message and process. Consumers can use the
> > > same mechanism as they are doing now.
> > >
> > > It adds an additional feature to pulsar in which if a producer knows how
> > > many times an event should be processed a max apart from the consumer max
> > > retry and send to DLQ if reaches its limit. Then the above change will
> > > check if the limit is reached for reconsume later then it will send to
> > DLQ
> > > even if the consumer has higher reconsme later retry limit.
> > >
> > > if producers do not pass the max retry limit it will obey the consumer
> > > retry limit.
> > >
> > > Also in this PIP we are not removing the consumer own retry limit it
> > would
> > > be there. but get overridden by message limit if passed by the producer.
> > if
> > > a consumer wants to re-override that limit ( temporarily change in the
> > > retry limit) which also can be achieved by sending a new max retry as
> > > custom property to the `reconsumeLater` function.
> > >
> > > So let me rephrase the subject for this pip is to `*MAX RECONSUME TIMES
> > > per Message*`.
> > >
> > > Michael's point about adding relation between producer and consumer?
> > >
> > > Yeah, It added a relation b/w them but it would be loose coupling.
> > Because
> > > producers can send the max retry limit in a message which can be
> > overridden
> > > by the consumer if needed. so consumers do not have to obey the limit if
> > > needed.
> > >
> > > Why I prefer message properties over `MessageMetadata` is because if a
> > > consumer wants to override the max limit (temporarily or permanent) then
> > it
> > > would be difficult to update if required.
> > >
> > > I hope the above content answers most of the doubts about this PIP.
> > >
> > > On Tue, Jan 10, 2023 at 5:59 PM r...@apache.org > >
> > > wrote:
> > >
> > >> Thanks for submitting this PIP, Nitin Goyal.
> > >>
> > >> Seeing that you have submitted a related pull request in the Go SDK
> > >>
Re: [ANNOUNCE] New Committer: Baodi Shi
Congrats > 2023年1月18日 21:35，Yunze Xu 写道： > > The Project Management Committee (PMC) for Apache Pulsar has invited > Baodi Shi (https://github.com/shibd) to become a committer and we are > pleased to announce that he has accepted. > > Being a committer enables easier contribution to the project since > there is no need to go via the patch submission process. This should > enable better productivity. > > Welcome and congratulations, Baodi Shi! > > Please join us in congratulating and welcoming Baodi Shi onboard! > > Thanks, > Yunze on behalf of the Pulsar PMC
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congrats！ > 2023年1月18日 21:50，PengHui Li 写道： > > Hi all, > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > (https://github.com/congbobo184) as a member of the PMC and we are > pleased to announce that he has accepted. > > He is very active in the community in the past few years and made a lot of > great contributions > such as transactions and schemas. > > Welcome Bo Cong to the Apache Pulsar PMC. > > Best Regards, > Penghui on behalf of the Pulsar PMC
Re: [DISCUSS] Make the default value of param "--get-subscription-backlog-size" of admin API "topics stats" true
+1
Haiting
On Tue, Jan 17, 2023 at 5:50 PM Yubiao Feng
wrote:
>
> Hi Asaf
>
> > It's worth noting that the estimated backlog size of a subscription is
> estimated since it doesn't consider any acknowledged messages between the
> mark delete position and the last message. It simply assumes all messages
> between the mark delete position and the last message have not been
> acknowledged.
>
> Yes, it's not the exact value of the backlog. There are two reasons for the
> loss of accuracy:
> - Whether the Entry size is closer to the `averageSize`.
> - The number of messages after the mark deleted position has been
> acknowledged.
>
> Thanks
> Yubiao Feng
>
> On Tue, Jan 17, 2023 at 3:31 PM Asaf Mesika wrote:
>
> > Small question regarding this:
> >
> > The code for calculation is:
> >
> > long estimateBacklogFromPosition(PositionImpl pos) {
> > synchronized (this) {
> > long sizeBeforePosLedger =
> > ledgers.headMap(pos.getLedgerId()).values()
> > .stream().mapToLong(LedgerInfo::getSize).sum();
> > LedgerInfo ledgerInfo = ledgers.get(pos.getLedgerId());
> > long sizeAfter = getTotalSize() - sizeBeforePosLedger;
> > if (ledgerInfo == null) {
> > return sizeAfter;
> > } else if (pos.getLedgerId() == currentLedger.getId()) {
> > return sizeAfter - consumedLedgerSize(currentLedgerSize,
> > currentLedgerEntries, pos.getEntryId());
> > } else {
> > return sizeAfter -
> > consumedLedgerSize(ledgerInfo.getSize(), ledgerInfo.getEntries(),
> > pos.getEntryId());
> > }
> > }
> > }
> >
> > and
> >
> > private long consumedLedgerSize(long ledgerSize, long ledgerEntries,
> > long consumedEntries) {
> > if (ledgerEntries <= 0) {
> > return 0;
> > }
> > if (ledgerEntries <= (consumedEntries + 1)) {
> > return ledgerSize;
> > } else {
> > long averageSize = ledgerSize / ledgerEntries;
> > return consumedEntries >= 0 ? (consumedEntries + 1) * averageSize
> > : 0;
> > }
> > }
> >
> >
> >
> > It's worth noting that the estimated backlog size of a subscription is
> > estimated since it doesn't consider any acknowledged messages between the
> > mark delete position and the last message. It simply assumes all messages
> > between the mark delete position and the last message have not been
> > acknowledged.
> >
> > Good idea - +1
> >
> > On Tue, Jan 17, 2023 at 4:12 AM PengHui Li
> > wrote:
> >
> > > +1
> > >
> > > Penghui
> > >
> > > > On Jan 16, 2023, at 23:36, Yubiao Feng > .INVALID>
> > > wrote:
> > > >
> > > > Hi community
> > > >
> > > > I am starting a DISCUSS for making the default value of the parameter
> > > > "--get-subscription-backlog-size" of admin API "topics stats" true.
> > > >
> > > > In the PR https://github.com/apache/pulsar/pull/9302, the property
> > > backlog
> > > > size of each subscription returned in the response of the API topics
> > > stats,
> > > > by default this property is always equal to 0 in response, and this
> > will
> > > > confuse users. Since the calculation of backlog size is done in broker
> > > > memory, there is no significant overhead(the process is described in
> > the
> > > > following section), so I think the correct values should be displayed
> > by
> > > > default.
> > > >
> > > > ### The following two APIs should be affected:
> > > >
> > > > In Pulsar admin API
> > > > ```
> > > > pulsar-admin topics stats persistent://test-tenant/ns1/tp1
> > > > --get-subscription-backlog-size
> > > > pulsar-admin topics stats persistent://test-tenant/ns1/tp1 -sbs
> > > > ```
> > > > the default value of parameter `--get-subscription-backlog-size` will
> > be
> > > > `true`
> > > >
> > > > In Pulsar Rest API
> > > > ```
> > > > curl GET "http://127.0.0.1:8080/test-tenant/ns1/tp1/stats
> > > > "?subscriptionBacklogSize=true
> > > > ```
> > > > the default value of parameter `subscriptionBacklogSize ` will be
> > `true`
> > > >
> > > >
> > > > ### The following is the process of calculating backlog size:
> > > > - Divide `PersistentTopc.ledgers` into two parts according to the
> > > ledgerId
> > > > of the mark delete position of the cursor. The second part is ledgers
> > > > indicating the messages still need to be consumed, aka
> > > backlogSizeInLedgers.
> > > > - Find the LedgerInfo whose ledgerId is the same as the ledgerId of the
> > > > mark delete position of the cursor, and we can also divide the ledger
> > > into
> > > > two parts, the second part is entries indicating the messages still
> > need
> > > to
> > > > be consumed, multiply the average size of each entry in metrics by the
> > > > number of still need to be consumed entries we can get the backlog size
> > > in
> > > > this ledger. aka backlogSizeInEntries.
> > > > - `backlogSizeInLe dgers` + `backlogSizeInEntries`
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > >
> > >
> >
Re: [DISCUSS] PIP-240 A new API to unload subscriptions
I agree with Penghui & Xiaolong, 1. Restarting a service is usually the most common and effective option for service maintainers to recover a service and minimize the business loss. With this subscription unloading, we can reduce the impact significantly, as unloading topics will affect message writing, which has much more influence for online business. 2. Having this subscription doesn't conflict with solving the real issue. Like broker restarting, it just can buy us more time to locate the real problem. BR, Haiting On Thu, Jan 19, 2023 at 11:42 AM r...@apache.org wrote: > > Hello Joe and Enrico: > > I agree with what you've been emphasizing that we need to fix these issues > at the root cause. During the maintenance of the Go SDK, we have > encountered many stuck problems since version 0.4.0, some of which belonged > to the logic errors handled by the Go SDK itself, and some of which were > caused by the user's wrong use of the Go SDK, until the previous 0.8 .0 > version, the Go SDK is used on a large scale in our environment. In the > iterations of these versions, we have been trying to completely fix these > BUGs. This is what our maintainers have been working hard on and it is also > a final form we expect Pulsar - everything looks OK. > > However, during the iteration of the Go SDK version from 0.4.0 to 0.8.0, > users of our production environment encountered similar problems many > times. Again, for a user in a production environment, for example, the > current user encounters a situation where consumption is blocked. The user > finds you and expects us to use some means to quickly allow consumers to > continue to consume news? Or do we keep users in the production environment > in a stuck state until we find the root cause of the problem and fix it for > users, pushing users to upgrade. I think everyone's answer tends to be the > latter. We will not directly expose the hack operations of unload topic and > unload sub to users, but to Pulsar's operation and maintenance personnel, > so it is more like an operation and maintenance tool , rather than the > interface called by the user. So I think this impact is controllable for > Pulsar as a whole, which is why I support it. > > Again, this PIP is more about buying more time for us to locate the problem > while minimizing the impact on production users. It’s not that with this > interface we don’t locate the real causes of the stuck. On the contrary, we > are making more trade-offs between users and positioning issues, buying us > more time for positioning issues. > > -- > Thanks > xiaolong ran > > PengHui Li 于2023年1月18日周三 11:48写道： > > > > What kind of problems is this trying to fix? > > And why cannot that be solved by client-side fixes? > > > > Yes, most of the issue is from the client side, rarely from the broker. > > But the application also needs time to fix the issue to release and deploy > > the fix > > to the production environment. Unloading the subscription is just a > > temporary > > way to mitigate the issue and reduce the impact. It will not fix the issue > > completely. > > > > What I learned is to capture the heap dump, topics stats, internal stats, > > and logs from the broker and client and then try to unload the topic to > > see if the problem is mitigated. If not, then try to restart the broker or > > client, > > most of the time, the problem can be mitigated in this way. > > Then we can continue to reproduce the issue and investigate the issue > > from the captured heap dump and logs. > > > > > In shared sub issues, it's hard to pinpoint which consumer/where > > the problem lies, and to reset that one at the client. The totality of > > state spread between the brokers and all the consumers of the shared sub > > needs to be put together . Is that why we are doing this? > > > > From my experience, most are from Shared and key shared subscriptions. > > Most of the issues come from misuse, rarely from the BUGs of brokers or > > clients. > > > > Regards, > > Penghui > > > > > > On Wed, Jan 18, 2023 at 11:31 AM Joe F wrote: > > > > > Inclined to agree with Enrico. If it's a hard problem, it will repeat, > > and > > > this is not helping. If it's some race on the client, it will occur > > > randomly and rarely, and this unload sub will get programmed in as a way > > of > > > life. > > > > > > >If you don't think unloading the subscription can't help anything. > > > Unloading > > > the topic should be the same. From my experience, most of the unloading > > > topic operations are to mitigate the problems related to message > > > consumption. > > > > > > Comparisons with unloading a topic are not the bar here, as that is a > > first > > > class broker utility that is needed for operational reasons outside of > > > "fixing" consumer side issues . The side effect of using "unload topic" > > is > > > a loss of transient topic state. I will fully agree that this side-effect > > > has been pervasively abused for fixing problems (ala
Re: [ANNOUNCE] New Committer: Baodi Shi
congratulation! On 2023/01/18 13:35:41 Yunze Xu wrote: > The Project Management Committee (PMC) for Apache Pulsar has invited > Baodi Shi (https://github.com/shibd) to become a committer and we are > pleased to announce that he has accepted. > > Being a committer enables easier contribution to the project since > there is no need to go via the patch submission process. This should > enable better productivity. > > Welcome and congratulations, Baodi Shi! > > Please join us in congratulating and welcoming Baodi Shi onboard! > > Thanks, > Yunze on behalf of the Pulsar PMC >
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
congratulation! On 2023/01/18 13:50:12 PengHui Li wrote: > Hi all, > > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong > (https://github.com/congbobo184) as a member of the PMC and we are > pleased to announce that he has accepted. > > He is very active in the community in the past few years and made a lot of > great contributions > such as transactions and schemas. > > Welcome Bo Cong to the Apache Pulsar PMC. > > Best Regards, > Penghui on behalf of the Pulsar PMC >
Re: [ANNOUNCE] New Committer: Baodi Shi
Congratulations! Best regards Haiting On Thu, Jan 19, 2023 at 1:53 PM Zixuan Liu wrote: > > Congrats! Baodi > > Best, > Zixuan > > ZhangJian He 于2023年1月19日周四 12:34写道： > > > Congratulations > > > > Thanks > > ZhangJian He > > > > > > On Thu, 19 Jan 2023 at 11:47, r...@apache.org > > wrote: > > > > > Congratulations!! > > > > > > -- > > > Thanks > > > Xiaolong Ran > > > > > > tison 于2023年1月19日周四 10:07写道： > > > > > > > Congrats and well deserved! > > > > > > > > Welcome onboard, Baodi :) > > > > > > > > Best, > > > > tison. > > > > > > > > > > > > Kai Wang 于2023年1月19日周四 09:57写道： > > > > > > > > > Congratulations! > > > > > > > > > > Thanks, > > > > > Kai > > > > > On Jan 18, 2023 at 9:36 PM +0800, dev@pulsar.apache.org, wrote: > > > > > > > > > > > > Congratulations ! > > > > > > > > > > > > > >
Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar
Congrats! Bo Best, Haiting On Thu, Jan 19, 2023 at 2:35 PM Xiangying Meng wrote: > > Congratulations > > Thanks > Xiangying > > On Thu, Jan 19, 2023 at 1:54 PM Zixuan Liu wrote: > > > Congrats! Bo > > > > Best, > > Zixuan > > > > ZhangJian He 于2023年1月19日周四 12:33写道： > > > > > Congratulations > > > > > > Thanks > > > ZhangJian He > > > > > > > > > On Thu, 19 Jan 2023 at 11:47, r...@apache.org > > > wrote: > > > > > > > Congratulations!!! > > > > > > > > -- > > > > Thanks > > > > Xiaolong Ran > > > > > > > > Baodi Shi 于2023年1月19日周四 10:08写道： > > > > > > > > > Congratulations ! > > > > > > > > > > Thanks, > > > > > Baodi Shi > > > > > > > > > > > > > > > 在 2023年1月19日 09:56:34 上，Kai Wang 写道： > > > > > > > > > > > Congratulations! > > > > > > > > > > > > Thanks, > > > > > > Kai > > > > > > On Jan 18, 2023 at 9:50 PM +0800, PengHui Li , > > > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo > > > > Cong > > > > > > > > > > > > (https://github.com/congbobo184) as a member of the PMC and we are > > > > > > > > > > > > pleased to announce that he has accepted. > > > > > > > > > > > > > > > > > > He is very active in the community in the past few years and made a > > > lot > > > > > of > > > > > > > > > > > > great contributions > > > > > > > > > > > > such as transactions and schemas. > > > > > > > > > > > > > > > > > > Welcome Bo Cong to the Apache Pulsar PMC. > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > Penghui on behalf of the Pulsar PMC > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Redundant ServiceUrlProvider design and improper use of PIP-121
Is it better to introduce a service URL detect interval to the service URL
provider?
I think the `updateServiceUrl` is not the initial purpose of exposing to
the Client API.
It looks like users just provide the interval of checking whether the
service URL is changed.
The Pulsar client will check it automatically. Using updateServiceUrl can
also achieve the purpose,
but users need to provide a fake service URL first or fetch the service URL
before
creating the client.
Another reason we need service URL provider API is that one team will
usually
provide an extra pulsar client lib with the service URL provider
implementation.
The lib
can be used across multiple teams. So from the user's perspective, they
only need to apply
a service URL provided to the client, they don't care about what is the
service URL and how to
update it.
Thanks,
Penghui
On Thu, Jan 19, 2023 at 3:43 PM Baodi Shi wrote:
> Hi, Yunze:
>
> Obviously, the `ServiceUrlProvider` config is redundant.
> >
>
> Agree. In fact, The client already provides the updateServiceUrl method,
> which the user can use to implement a dynamic update service URL. As for
> how the user implements it and how to close his resources, I think it can
> be left to the user.
>
>
> 在 2023年1月19日 15:16:52 上，Yunze Xu 写道：
>
> > Hi all,
> >
> > Currently we have a `ServiceUrlProvider` interface to configure when
> > constructing a PulsarClient in `ClientBuilder#serviceUrlProvider`.
> > From the beginning, I thought the `getServiceUrl` method is called
> > each time the service URL is used, e.g. topic metadata lookup.
> > However, the `getServiceUrl` method is only called when constructing
> > the PulsarClient object. To update the PulsarClient's internal service
> > URL, `PulsarClient#updateServiceUrl` must be called. Therefore, if we
> > want to implement a `ServiceUrlProvider` that retrieves the latest
> > service URL from a database, I have to implement it like:
> >
> > ```java
> > class DataBaseServiceUrlProvider implements ServiceUrlProvider {
> >
> >private final ScheduledExecutorService executor =
> > Executors.newSingleThreadScheduledExecutor();
> >
> >@Override
> >public void initialize(PulsarClient client) {
> >executor.schedule(() -> {
> >try {
> >client.updateServiceUrl(readServiceUrlFromDB()/* a
> > fake method */);
> >} catch (PulsarClientException e) {
> >throw new RuntimeException(e);
> >}
> >}, 1, TimeUnit.SECONDS);
> >}
> >
> >@Override
> >public String getServiceUrl() {
> >return "pulsar://localhost:6650";
> >}
> >
> >@Override
> >public void close() {
> >executor.shutdown();
> >}
> > }
> > ```
> >
> > The key point is, if we didn't call `client.updateServiceUrl` and only
> > modified the returned value of `getServiceUrl` periodically, the
> > internal service URL would never be updated.
> >
> > Based on the provider above, the following two code snippets could be
> > nearly the same.
> >
> > ```java
> > var client = PulsarClient.builder().serviceUrlProvider(new
> > DataBaseServiceUrlProvider()).build();
> > /* ... */
> > client.close();
> > ```
> >
> > ```java
> > var client =
> > PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
> > var provider = new DataBaseServiceUrlProvider();
> > provider.initialize(client);
> > /* ... */
> > provider.close();
> > client.close();
> > ```
> >
> > Obviously, the `ServiceUrlProvider` config is redundant.
> >
> > PIP-121 implements the `AutoClusterFailover` as the service URL
> > provider. However, it also calls the following methods periodically:
> > - PulsarClientImpl#updateAuthentication
> > - PulsarClientImpl#updateTlsTrustCertsFilePath
> >
> > It's unnatural and intuitive to say a service URL provider could
> > modify the internal states of `PulsarClient`, including:
> > - the service URL
> > - the authentication
> > - the TLS trust certificate file
> > - ...
> >
> > BTW, the implementation of PIP-121 [1] is different from the design [2].
> >
> > [1] https://github.com/apache/pulsar/pull/13316
> > [2] https://github.com/apache/pulsar/issues/13315
> >
> > Thanks,
> > Yunze
> >
>
