Re: [DISCUSS] Release Pulsar 2.9.4

[houxiaoyu]

+1

Best,
Xiaoyu Hou

丛搏  于2022年11月3日周四 14:01写道：

> Hello, Pulsar community:
>
> I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> three months since 2.9.3 was released.
>
> There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> directly into branch-2.9.
>
> There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> see if they will be completed soon or will need to be pushed to 2.9.5
>
> If you have any important fixes or any questions, please reply to this
> email, and we will evaluate whether to include them in 2.9.4
>
> Thanks,
> Bo
> [0] -
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> [1] -
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> I will
>

[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: struggling with pulsar functions in kubernetes

[GitBox]

GitHub user michaeljmarshall added a comment to the discussion: struggling with 
pulsar functions in kubernetes

@jesumyip - in your function worker config, do you have 
`brokerClientTrustCertsFilePath` or `tlsTrustCertsFilePath` set? Based on 
reading through the code, that configuration is used to read the bytes into the 
function worker's memory, create a kubernetes secret, then mount that secret 
into the function pod.

This appears to be documented here: 
https://pulsar.apache.org/docs/next/functions-worker-run-separately/#enable-tls-encryption.

Here are some of the relevant code references:

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java#L607-L615

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java#L774-L785

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactory.java#L221

(A couple skipped references, but it's easy to follow in an IDE.)

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/auth/KubernetesSecretsTokenAuthProvider.java#L279-L287

GitHub link: 
https://github.com/apache/pulsar/discussions/18332#discussioncomment-4053503


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: Releasing current master as Pulsar 2.11.0 ?

[Dave Fisher]

Inline

Sent from my iPhone

> On Nov 3, 2022, at 6:55 AM, Enrico Olivelli  wrote:
> 
> PengHui,
> 
>> Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li
>>  ha scritto:
>> 
>>> As it is, we already need to discuss EOL for 2.7 and 2.8.
>> 
>> Agree. We should clarify this one.
>> I think we can stop to provide new releases for 2.7
>> and only security or critical bugs for 2.8 (one more official release)
>> 
>> https://github.com/apache/pulsar/issues/15966 will make the
>> release strategy clear.
>> 
>> LTS -> 36 months (24 + 12)
>> Feature release -> 6 months (3+3)
>> 
>> Thanks,
>> Penghui
>> 
>> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall 
>> wrote:
>> 
>>> I am concerned that we have too many active release branches, and planning
>>> to follow 2.11.0 with 3.0.0 soon after feels like it will make that problem
>>> worse. As it is, we already need to discuss EOL for 2.7 and 2.8.
>>> 
>>> Thanks,
>>> Michael
>>> 
 On Mon, Oct 31, 2022 at 7:55 PM PengHui Li  wrote:
>>> 
 Releasing from the master branch will bring more uncertainty, no?
 We have fixed many regressions that were introduced to branch-2.11.
 If we cut a new branch-2.11 based on the master branch. Maybe new
 regressions
 will happen again. This may make us wait another month to have a 2.11.0
 release.
> 
> I am not sure.
> I don't know if anyone is actively testing the 2.11 branch more than
> the master branch.
> On my side the (automated) testing that I do with my colleagues on
> branch-2.11 is basically the same as for the master branch.
> 
> I believe that if we want to cut a 2.11 release that is not branched
> again from the master branch
> we really must start the release as soon as BK 4.15.3 is released

I understand that Bookkeeper issues have Ben what’s blocking 2.11
> 
> Many people contributed features to the master branch that cannot be
> shipped with 2.11 because
> they are considered "breaking changes".
> But 2.11 was supposed to be released in August, more than 3 months ago.

I think we can recognize that our past history has been that there are often 3 
or 4 RCs for our 2.x.0 releases.

Maybe we should be cherry picking some PRs on master to 2.11 before we start 
the process? It may or may not save an RC but it will give us time to be 
realistic about a reasonable cadence from 2.10.x to 2.11.x to 2.12.x … it’s 
hard to support many versions at once. The CVE announced today took months to 
be included in all of our current releases from 2.7.5 to 2.10.2. Separation of 
C++ and Pulsar client releases from Pulsar releases helps here, but it may not 
with the next security issue.

Regards,
Dave
> 
> 
> Enrico
> 
> 
 
 IMO, we can start Pulsar 3.0 (follow
 https://github.com/apache/pulsar/issues/15966)
 after 2.11.0 is released instead of waiting for 3 more months.
 
 For https://github.com/apache/bookkeeper/issues/3466
 I don't think it's a blocker for the Pulsar release for now.
 Yes, it is worth investigating more. We also tried a chaos test for that
 case.
 We haven't reproduced the problem on Pulsar.
 
 Now, we are just waiting for the new BookKeeper release 4.15.3 since
>>> 4.15.2
 has regressions [1]
 
 [1] https://github.com/apache/bookkeeper/pull/3523
 
 Thanks,
 Penghui
 
 On Tue, Nov 1, 2022 at 3:10 AM Michael Marshall 
 wrote:
 
> I have not followed the branch-2.11 work closely, but I think it makes
> sense to re-create branch-2.11 from the current master.
> 
> We created branch-2.11 almost 3 months ago. Re-creating the branch
> will prevent unnecessary delay on new features added over the past 3
> months.
> 
> If we follow through with this proposal, we will need to clean up PR
> tags and milestones to prevent confusion.
> 
> Thanks,
> Michael
> 
> On Mon, Oct 31, 2022 at 3:31 AM Enrico Olivelli 
> wrote:
>> 
>> Hello Pulsar fellows,
>> 
>> I think that too much time passed since we wanted to cut 2.11.
>> 
>> The branch-2.11 contains some code used by no one.
>> 
>> In the meantime many features went into master branch,
>> 
>> I don't think that it is worth it to cut a release from branch-2.11
>> and start with something that is already stale.
>> 
>> I propose to drop branch-2.11 and create a new branch out of the
>> current master branch and start the period of hardening before
>>> cutting
>> the release.
>> 
>> IIUC we are waiting for this BookKeeper issue to be confirmed or
>>> fixed
>> or closed as "not a problem":
>> https://github.com/apache/bookkeeper/issues/3466
>> I am personally working on that case together with the folks you
>> created the issue.
>> Honestly I have never been able to reproduce the problem with Pulsar.
>> I believe that it will take at least another week before having more
>> results about the investigations I am doing on 

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

[Michael Marshall]

Thank you for your detailed responses, Heesung.

> We are not planning to expose this feature to users
> soon unless demanded and proven to be stable.

In that case, I think we should move forward with this PIP. I have a
different opinion about the trade offs for the two designs, but none
of my concerns are problems that could not be solved later if we
encounter problems.

Just to say it succinctly, my concern is that broadcasting all
attempts to acquire ownership of every unclaimed bundle to all brokers
will generate a lot of unnecessary traffic.

> https://github.com/apache/pulsar/pull/18079/files#diff-7f9930a5c7896b411f61901cf38371e23ba69e753f460bf7f520f6f800d8321a

Thank you for this reference. I missed it. That is great documentation!

> In fact, with the `compact(T prev, T cur)` api only, it is not clear if
> prev => cur is a valid transition or not(if invalid, we should filter out
> the cur message instead of further compacting/merging). I think we still
> need to keep the `isValid()` and `merge()` separated.

I was thinking that the result of `compact` would be the result put in
the table view or written to the compacted topic. The one issue might
be about keeping the memory utilization down for use cases that are
not updating the message's value but are only selecting "left" or
"right". I thought we could infer when to keep the message id vs keep
the message value, but that might be easy to implement.

My final critique is that I think `isValid` could have a better name.
In the event this does become a public API, I don't think all use
cases will think about which event should be persisted in terms of
validity.

The main name that comes to my mind is `bool shouldKeepLeft(T prev, T
cur)`. When true, prev wins. When false, cur wins. That nomenclature
comes from Akka Streams. It's not perfect, but it is easy to infer
what the result will do.

> Regarding redundant deserialization, the input type `T` is the type of
> message value, so the input values are already deserialized.

Great, I should have realized that. That takes care of that concern.

Thanks,
Michael

On Thu, Nov 3, 2022 at 7:37 PM Heesung Sohn
 wrote:
>
> Hi,
>
> I have a different thought about my previous comment.
>
> - Agreed with your point that we should merge CompactionStrategy APIs. I
> updated the interface proposal in the PIP. I replaced `"isValid",
> "isMergeEnabled", and "merge"` apis with "compact" api.
>
> boolean isValid(T prev, T cur)
> boolean isMergeEnabled()
> T merge(T prev, T cur)
>
> =>
>
> T compact(T prev, T cur)
>
> In fact, with the `compact(T prev, T cur)` api only, it is not clear if
> prev => cur is a valid transition or not(if invalid, we should filter out
> the cur message instead of further compacting/merging). I think we still
> need to keep the `isValid()` and `merge()` separated.
>
> Regarding redundant deserialization, the input type `T` is the type of
> message value, so the input values are already deserialized. We don't want
> to expose the Message interface in this CompactionStrategy to avoid
> message serialization/deserialization dependencies in the
> CompactionStrategy.
>
> The `merge()` functionality is suggested for more complex use cases (merge
> values instead of just filtering), and to support this `merge()`, we need
> to internally create a new msg with the compacted value, metadata, and
> messageId copies. We could initially define `isValid()` only in
> CompactionStrategy, and add `isMergeEnabled() and merge()` later in the
> CompactionStrategy interface if requested.
>
> Regards,
> Heesung
>
>
> On Thu, Nov 3, 2022 at 9:50 AM Heesung Sohn 
> wrote:
>
> > Oops! Michael, I apologize for the typo in your name.
> >
> > On Thu, Nov 3, 2022 at 9:47 AM Heesung Sohn 
> > wrote:
> >
> >> Hi Machel,
> >>
> >> Here are my additional comments regarding your earlier email.
> >>
> >> - I updated the PIP title to show that this will impact table view as
> >> well.
> >>
> >> - PIP-192 : https://github.com/apache/pulsar/issues/16691 shows the
> >> general idea of the states and their actions, and I defined the actual
> >> states here in the PR,
> >> https://github.com/apache/pulsar/pull/18079/files#diff-7f9930a5c7896b411f61901cf38371e23ba69e753f460bf7f520f6f800d8321a.
> >>  I
> >> will further clarify the bundle state data validation logic when
> >> introducing `BundleStateCompactionStrategy` class. This PIP is to support
> >> CompactionStrategy in general.
> >>
> >> - Agreed with your point that we should merge CompactionStrategy APIs. I
> >> updated the interface proposal in the PIP. I replaced `"isValid",
> >> "isMergeEnabled", and "merge"` apis with "compact" api.
> >>
> >>
> >> Thanks,
> >> Heesung
> >>
> >>
> >> On Tue, Nov 1, 2022 at 11:26 AM Heesung Sohn <
> >> heesung.s...@streamnative.io> wrote:
> >>
> >>> Hi,
> >>> Thank you for the great comments.
> >>> Please find my comments inline too.
> >>>
> >>> Regards,
> >>> Heesung
> >>>
> >>> On Mon, Oct 31, 2022 at 10:59 PM Michael Marshall 
> >>> 

Re: CVE-2022-33684: Apache Pulsar: Disabled Certificate Validation for OAuth Client Credential Requests makes C++/Python Clients vulnerable to MITM attack

[Yunze Xu]

Hi Michael,

Thanks for your explanation. I think now we can mark #16064 [1] as the
fix to the CVE.

In addition, the Apache Node.js client [2] is also based on the C++
client. I'm not familiar
with how to install the Node.js client at the moment. It looks like it
requires a pre-installation
of the C++ client. So for Node.js users, they must upgrade the C++
client to the proper version.

[1] https://github.com/apache/pulsar/pull/16064
[2] https://github.com/apache/pulsar-client-node

Thanks,
Yunze


On Fri, Nov 4, 2022 at 2:40 AM Michael Marshall  wrote:
>
> Severity: high
>
> Description:
>
> The Apache Pulsar C++ Client does not verify peer TLS certificates when 
> making HTTPS calls for the OAuth2.0 Client Credential Flow, even when 
> tlsAllowInsecureConnection is disabled via configuration. This vulnerability 
> allows an attacker to perform a man in the middle attack and intercept and/or 
> modify the GET request that is sent to the ClientCredentialFlow 'issuer url'. 
> The intercepted credentials can be used to acquire authentication data from 
> the OAuth2.0 server to then authenticate with an Apache Pulsar cluster.
>
> An attacker can only take advantage of this vulnerability by taking control 
> of a machine 'between' the client and the server. The attacker must then 
> actively manipulate traffic to perform the attack.
>
> The Apache Pulsar Python Client wraps the C++ client, so it is also 
> vulnerable in the same way.
>
> This issue affects Apache Pulsar C++ Client and Python Client versions 2.7.0 
> to 2.7.4; 2.8.0 to 2.8.3; 2.9.0 to 2.9.2; 2.10.0 to 2.10.1; 2.6.4 and earlier.
>
> Mitigation:
>
> Any users running affected versions of the C++ Client or the Python Client 
> should rotate vulnerable OAuth2.0 credentials, including client_id and 
> client_secret.
>
> 2.7 C++ and Python Client users should upgrade to 2.7.5 and rotate vulnerable 
> OAuth2.0 credentials.
> 2.8 C++ and Python Client users should upgrade to 2.8.4 and rotate vulnerable 
> OAuth2.0 credentials.
> 2.9 C++ and Python Client users should upgrade to 2.9.3 and rotate vulnerable 
> OAuth2.0 credentials.
> 2.10 C++ and Python Client users should upgrade to 2.10.2 and rotate 
> vulnerable OAuth2.0 credentials.
> 3.0 C++ users are unaffected and 3.0 Python Client users will be unaffected 
> when it is released.
> Any users running the C++ and Python Client for 2.6 or less should upgrade to 
> one of the above patched versions.
>
> Credit:
>
> This issue was discovered by Michael Rowley, michaellrow...@protonmail.com
>

Re: [DISCUSS] Release Pulsar 2.9.4

[Michael Marshall]

+1 - sounds good to me. Thanks for volunteering to be the release manager Bo.

Thanks,
Michael

On Thu, Nov 3, 2022 at 11:10 PM Zike Yang  wrote:
>
> +1
>
> Best,
> Zike Yang
>
> On Fri, Nov 4, 2022 at 10:28 AM Max Xu  wrote:
> >
> > +1
> >
> > Best,
> > Max Xu
> >
> >
> > On Thu, Nov 3, 2022 at 2:01 PM 丛搏  wrote:
> >
> > > Hello, Pulsar community:
> > >
> > > I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> > > three months since 2.9.3 was released.
> > >
> > > There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> > > cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> > > directly into branch-2.9.
> > >
> > > There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> > > see if they will be completed soon or will need to be pushed to 2.9.5
> > >
> > > If you have any important fixes or any questions, please reply to this
> > > email, and we will evaluate whether to include them in 2.9.4
> > >
> > > Thanks,
> > > Bo
> > > [0] -
> > > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> > > [1] -
> > > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> > > I will
> > >

[ANNOUNCE] Apache Pulsar Helm Chart version 3.0.0 Released

[Michael Marshall]

Dear Pulsar community,

I am pleased to announce that we have released Apache Pulsar Helm
chart 3.0.0. It ships with Apache Puslar version 2.10.2, our latest
release.

The source release, as well as the "binary" Helm Chart release, are available:

Official Sources: https://pulsar.apache.org/download/
ArtifactHub: https://artifacthub.io/packages/helm/apache/pulsar/3.0.0
Docs: https://pulsar.apache.org/docs/helm-overview
Release Notes: 
https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-3.0.0

Thanks to all the contributors who made this possible.

Regards,
The Apache Pulsar Team

Re: [DISCUSS] Release Pulsar 2.9.4

[Zike Yang]

+1

Best,
Zike Yang

On Fri, Nov 4, 2022 at 10:28 AM Max Xu  wrote:
>
> +1
>
> Best,
> Max Xu
>
>
> On Thu, Nov 3, 2022 at 2:01 PM 丛搏  wrote:
>
> > Hello, Pulsar community:
> >
> > I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> > three months since 2.9.3 was released.
> >
> > There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> > cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> > directly into branch-2.9.
> >
> > There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> > see if they will be completed soon or will need to be pushed to 2.9.5
> >
> > If you have any important fixes or any questions, please reply to this
> > email, and we will evaluate whether to include them in 2.9.4
> >
> > Thanks,
> > Bo
> > [0] -
> > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> > [1] -
> > https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> > I will
> >

[DISCUSS] PIP-221: Make TableView support read the non-persistent topic

[Kai Wang]

Hi, pulsar-dev community,

I’ve opened a PIP to discuss : PIP-221: Make TableView support read the 
non-persistent topic.

PIP link: https://github.com/apache/pulsar/issues/18229

Thanks,
Kai

Re: [DISCUSS] Release Pulsar 2.9.4

[Max Xu]

+1

Best,
Max Xu


On Thu, Nov 3, 2022 at 2:01 PM 丛搏  wrote:

> Hello, Pulsar community:
>
> I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> three months since 2.9.3 was released.
>
> There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> directly into branch-2.9.
>
> There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> see if they will be completed soon or will need to be pushed to 2.9.5
>
> If you have any important fixes or any questions, please reply to this
> email, and we will evaluate whether to include them in 2.9.4
>
> Thanks,
> Bo
> [0] -
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> [1] -
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> I will
>

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

[Heesung Sohn]

Hi,

I have a different thought about my previous comment.

- Agreed with your point that we should merge CompactionStrategy APIs. I
updated the interface proposal in the PIP. I replaced `"isValid",
"isMergeEnabled", and "merge"` apis with "compact" api.

boolean isValid(T prev, T cur)
boolean isMergeEnabled()
T merge(T prev, T cur)

=>

T compact(T prev, T cur)

In fact, with the `compact(T prev, T cur)` api only, it is not clear if
prev => cur is a valid transition or not(if invalid, we should filter out
the cur message instead of further compacting/merging). I think we still
need to keep the `isValid()` and `merge()` separated.

Regarding redundant deserialization, the input type `T` is the type of
message value, so the input values are already deserialized. We don't want
to expose the Message interface in this CompactionStrategy to avoid
message serialization/deserialization dependencies in the
CompactionStrategy.

The `merge()` functionality is suggested for more complex use cases (merge
values instead of just filtering), and to support this `merge()`, we need
to internally create a new msg with the compacted value, metadata, and
messageId copies. We could initially define `isValid()` only in
CompactionStrategy, and add `isMergeEnabled() and merge()` later in the
CompactionStrategy interface if requested.

Regards,
Heesung


On Thu, Nov 3, 2022 at 9:50 AM Heesung Sohn 
wrote:

> Oops! Michael, I apologize for the typo in your name.
>
> On Thu, Nov 3, 2022 at 9:47 AM Heesung Sohn 
> wrote:
>
>> Hi Machel,
>>
>> Here are my additional comments regarding your earlier email.
>>
>> - I updated the PIP title to show that this will impact table view as
>> well.
>>
>> - PIP-192 : https://github.com/apache/pulsar/issues/16691 shows the
>> general idea of the states and their actions, and I defined the actual
>> states here in the PR,
>> https://github.com/apache/pulsar/pull/18079/files#diff-7f9930a5c7896b411f61901cf38371e23ba69e753f460bf7f520f6f800d8321a.
>>  I
>> will further clarify the bundle state data validation logic when
>> introducing `BundleStateCompactionStrategy` class. This PIP is to support
>> CompactionStrategy in general.
>>
>> - Agreed with your point that we should merge CompactionStrategy APIs. I
>> updated the interface proposal in the PIP. I replaced `"isValid",
>> "isMergeEnabled", and "merge"` apis with "compact" api.
>>
>>
>> Thanks,
>> Heesung
>>
>>
>> On Tue, Nov 1, 2022 at 11:26 AM Heesung Sohn <
>> heesung.s...@streamnative.io> wrote:
>>
>>> Hi,
>>> Thank you for the great comments.
>>> Please find my comments inline too.
>>>
>>> Regards,
>>> Heesung
>>>
>>> On Mon, Oct 31, 2022 at 10:59 PM Michael Marshall 
>>> wrote:
>>>
 > I think we lose a single linearized view.

 Which linearized view are we losing, and what is the role of that
 linearized view? I think I might be missing why it is important. I
 agree that consumers won't know about each unsuccessful attempted
 acquisition of a bundle, but that seems like unnecessary information
 to broadcast to every broker in the cluster.


>>>
>>> PIP-192 proposes an assignment, transfer, and split protocol(multi-phase
>>> state changes), relying on early broadcast across brokers, and all brokers
>>> react to their clients according to the state change notifications --
>>> brokers could defer any client lookups for bundle x if an
>>> assignment/transfer/split is ongoing for x(broadcasted early in the topic).
>>> One early broadcast example is the one that I discussed above, `When the
>>> topic broadcast is faster than the concurrent assignment requests.` I think
>>> the prefilter could delay this early broadcast, as it needs to go through
>>> the additional single-leader compaction path.
>>>
>>> The bundle state recovery process is simpler by a single linearized view.
>>>
>>> The single linearized view can be easier to debug bundle states. We can
>>> more easily track where the assignment requests come from and how it is
>>> compacted in a single linearized view.
>>>
>>>
>>>
 > I think the leader requires a write-through cache to compact messages
 based
 > on the latest states.

>>> This brings up an important point that I would like to clarify. If we
 trust the write ahead log as the source of truth, what happens when a
 bundle has been validly owned by multiple brokers? As a broker starts
 and consumes from the compacted topic, how do we prevent it from
 incorrectly thinking that it owns a bundle for some short time period
 in the case that the ownership topic hasn't yet been compacted to
 remove old ownership state?

>>>
>>> Since the multi-phase transfer protocol involves the source and
>>> destination broker's actions, the successful transfer should get the source
>>> and destination broker to have the (near) latest state. For example, if
>>> some brokers have old ownership states(network partitioned or delayed),
>>> they will redirect clients to the 

Re: [VOTE] Release Apache Pulsar Helm Chart 3.0.0 based on 3.0.0-candidate-1

[Michael Marshall]

The vote to release Apache Pulsar Helm Chart version 3.0.0 based on
3.0.0-candidate-1 is now closed.

The vote PASSED with 3 binding "+1", 0 non-binding "+1" and 0 "-1" votes:

"+1" Binding votes:

  - Enrico
  - Michael
  - Dave

I'll continue with the release process and the release announcement
will follow shortly.

Thanks,
Michael

On Thu, Nov 3, 2022 at 9:56 AM Dave Fisher  wrote:
>
> +1 (binding)
>
> I checked the signatures and checksums of the source and convenience binary 
> packages.
> I did a license check of both and got reasonable results. The NOTICE is fine 
> as well.
>
> I reviewed the release notes. Very important highlights, great job!
>
> I notice that on release the index.yaml will need to change to point to the 
> proper download url for apache.org. That is expected.
>
> Best Regards,
> Dave
>
>
> > On Oct 20, 2022, at 10:58 PM, Michael Marshall  wrote:
> >
> > Hello Apache Pulsar Community,
> >
> > This is a call for the vote to release the Apache Pulsar Helm Chart
> > version 3.0.0.
> >
> > This is our first official vote to release the helm chart. It follows
> > a newly merged release process that is still subject to change
> > (https://github.com/apache/pulsar-helm-chart/pull/301).
> >
> > I did a major version bump to mitigate the security concerns raised by
> > https://github.com/apache/pulsar-helm-chart/issues/294.
> >
> > The release candidate is available at:
> > https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.0.0-candidate-1/
> >
> > pulsar-chart-3.0.0-source.tar.gz - is the "main source release".
> > pulsar-3.0.0.tgz - is the binary Helm Chart release.
> >
> > Public keys are available at: https://www.apache.org/dist/pulsar/KEYS
> >
> > Here is a draft of the 3.0.0 release notes:
> > https://github.com/apache/pulsar-helm-chart/pull/322
> >
> > For convenience "index.yaml" has been uploaded (though excluded from
> > voting), so you can also run the below commands.
> >
> > helm repo add apache-pulsar-dist-dev
> > https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.0.0-candidate-1/
> > helm repo update
> > helm install pulsar apache-pulsar-dist-dev/pulsar
> >
> > pulsar-3.0.0.tgz.prov - is also uploaded for verifying Chart
> > Integrity, though it is not strictly required for releasing the
> > artifact based on ASF Guidelines.
> >
> > You can optionally verify this file using this helm plugin
> > https://github.com/technosophos/helm-gpg, or by using helm --verify
> > (https://helm.sh/docs/helm/helm_verify/). However, I had trouble with
> > the formatting of our KEYS file while trying to get the helm --verify
> > feature to work.
> >
> > helm gpg verify pulsar-3.0.0.tgz
> >
> > The vote will be open for at least 72 hours.
> >
> > Only votes from PMC members are binding, but members of the community are
> > encouraged to test the release and vote with "(non-binding)".
> >
> > For license checks, the .rat-excludes files is included, so you can
> > run the following to verify licenses (just update $PATH_TO_RAT to
> > point to your binary download of
> > https://creadur.apache.org/rat/download_rat.cgi):
> >
> > tar -xvf pulsar-chart-3.0.0-source.tar.gz
> > cd pulsar-chart-3.0.0
> > java -jar $PATH_TO_RAT/apache-rat-0.15/apache-rat-0.15.jar . -E 
> > .rat-excludes
> >
> > Additionally, tests to verify upgrade scenarios are particularly helpful.
> >
> > Please note that the version number excludes the `-candidate-X`
> > string, so it's now
> > simply 3.0.0. This will allow us to rename the artifact without modifying
> > the artifact checksums when we actually release it.
> >
> > Thanks,
> > Michael
>

Re: [DISCUSS] Release Pulsar 2.9.4

[Enrico Olivelli]

+1

Enrico

Il Gio 3 Nov 2022, 18:36 Yubiao Feng 
ha scritto:

> +1
>
> Thanks
> Yubiao
>
> On Thu, Nov 3, 2022 at 2:01 PM 丛搏  wrote:
>
> > Hello, Pulsar community:
> >
> > I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> > three months since 2.9.3 was released.
> >
> > There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> > cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> > directly into branch-2.9.
> >
> > There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> > see if they will be completed soon or will need to be pushed to 2.9.5
> >
> > If you have any important fixes or any questions, please reply to this
> > email, and we will evaluate whether to include them in 2.9.4
> >
> > Thanks,
> > Bo
> > [0] -
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> > [1] -
> >
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> > I will
> >
>

CVE-2022-33684: Apache Pulsar: Disabled Certificate Validation for OAuth Client Credential Requests makes C++/Python Clients vulnerable to MITM attack

[Michael Marshall]

Severity: high

Description:

The Apache Pulsar C++ Client does not verify peer TLS certificates when making 
HTTPS calls for the OAuth2.0 Client Credential Flow, even when 
tlsAllowInsecureConnection is disabled via configuration. This vulnerability 
allows an attacker to perform a man in the middle attack and intercept and/or 
modify the GET request that is sent to the ClientCredentialFlow 'issuer url'. 
The intercepted credentials can be used to acquire authentication data from the 
OAuth2.0 server to then authenticate with an Apache Pulsar cluster.

An attacker can only take advantage of this vulnerability by taking control of 
a machine 'between' the client and the server. The attacker must then actively 
manipulate traffic to perform the attack.

The Apache Pulsar Python Client wraps the C++ client, so it is also vulnerable 
in the same way.

This issue affects Apache Pulsar C++ Client and Python Client versions 2.7.0 to 
2.7.4; 2.8.0 to 2.8.3; 2.9.0 to 2.9.2; 2.10.0 to 2.10.1; 2.6.4 and earlier.

Mitigation:

Any users running affected versions of the C++ Client or the Python Client 
should rotate vulnerable OAuth2.0 credentials, including client_id and 
client_secret.

2.7 C++ and Python Client users should upgrade to 2.7.5 and rotate vulnerable 
OAuth2.0 credentials.
2.8 C++ and Python Client users should upgrade to 2.8.4 and rotate vulnerable 
OAuth2.0 credentials.
2.9 C++ and Python Client users should upgrade to 2.9.3 and rotate vulnerable 
OAuth2.0 credentials.
2.10 C++ and Python Client users should upgrade to 2.10.2 and rotate vulnerable 
OAuth2.0 credentials.
3.0 C++ users are unaffected and 3.0 Python Client users will be unaffected 
when it is released.
Any users running the C++ and Python Client for 2.6 or less should upgrade to 
one of the above patched versions.

Credit:

This issue was discovered by Michael Rowley, michaellrow...@protonmail.com

Re: [DISCUSS] Release Pulsar 2.9.4

[Yubiao Feng]

+1

Thanks
Yubiao

On Thu, Nov 3, 2022 at 2:01 PM 丛搏  wrote:

> Hello, Pulsar community:
>
> I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> three months since 2.9.3 was released.
>
> There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> directly into branch-2.9.
>
> There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> see if they will be completed soon or will need to be pushed to 2.9.5
>
> If you have any important fixes or any questions, please reply to this
> email, and we will evaluate whether to include them in 2.9.4
>
> Thanks,
> Bo
> [0] -
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> [1] -
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> I will
>

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

[Heesung Sohn]

Oops! Michael, I apologize for the typo in your name.

On Thu, Nov 3, 2022 at 9:47 AM Heesung Sohn 
wrote:

> Hi Machel,
>
> Here are my additional comments regarding your earlier email.
>
> - I updated the PIP title to show that this will impact table view as well.
>
> - PIP-192 : https://github.com/apache/pulsar/issues/16691 shows the
> general idea of the states and their actions, and I defined the actual
> states here in the PR,
> https://github.com/apache/pulsar/pull/18079/files#diff-7f9930a5c7896b411f61901cf38371e23ba69e753f460bf7f520f6f800d8321a.
>  I
> will further clarify the bundle state data validation logic when
> introducing `BundleStateCompactionStrategy` class. This PIP is to support
> CompactionStrategy in general.
>
> - Agreed with your point that we should merge CompactionStrategy APIs. I
> updated the interface proposal in the PIP. I replaced `"isValid",
> "isMergeEnabled", and "merge"` apis with "compact" api.
>
>
> Thanks,
> Heesung
>
>
> On Tue, Nov 1, 2022 at 11:26 AM Heesung Sohn 
> wrote:
>
>> Hi,
>> Thank you for the great comments.
>> Please find my comments inline too.
>>
>> Regards,
>> Heesung
>>
>> On Mon, Oct 31, 2022 at 10:59 PM Michael Marshall 
>> wrote:
>>
>>> > I think we lose a single linearized view.
>>>
>>> Which linearized view are we losing, and what is the role of that
>>> linearized view? I think I might be missing why it is important. I
>>> agree that consumers won't know about each unsuccessful attempted
>>> acquisition of a bundle, but that seems like unnecessary information
>>> to broadcast to every broker in the cluster.
>>>
>>>
>>
>> PIP-192 proposes an assignment, transfer, and split protocol(multi-phase
>> state changes), relying on early broadcast across brokers, and all brokers
>> react to their clients according to the state change notifications --
>> brokers could defer any client lookups for bundle x if an
>> assignment/transfer/split is ongoing for x(broadcasted early in the topic).
>> One early broadcast example is the one that I discussed above, `When the
>> topic broadcast is faster than the concurrent assignment requests.` I think
>> the prefilter could delay this early broadcast, as it needs to go through
>> the additional single-leader compaction path.
>>
>> The bundle state recovery process is simpler by a single linearized view.
>>
>> The single linearized view can be easier to debug bundle states. We can
>> more easily track where the assignment requests come from and how it is
>> compacted in a single linearized view.
>>
>>
>>
>>> > I think the leader requires a write-through cache to compact messages
>>> based
>>> > on the latest states.
>>>
>> This brings up an important point that I would like to clarify. If we
>>> trust the write ahead log as the source of truth, what happens when a
>>> bundle has been validly owned by multiple brokers? As a broker starts
>>> and consumes from the compacted topic, how do we prevent it from
>>> incorrectly thinking that it owns a bundle for some short time period
>>> in the case that the ownership topic hasn't yet been compacted to
>>> remove old ownership state?
>>>
>>
>> Since the multi-phase transfer protocol involves the source and
>> destination broker's actions, the successful transfer should get the source
>> and destination broker to have the (near) latest state. For example, if
>> some brokers have old ownership states(network partitioned or delayed),
>> they will redirect clients to the source(old) broker. However, by the
>> transfer protocol, the source broker should have the latest state, so it
>> can redirect the client again to the destination broker.
>>
>> When a broker restarts, it won't start until its BSC state to the (near)
>> latest (til the last known messageId at that time).
>>
>>
>>> > Pulsar guarantees "a single writer".
>>>
>>> I didn't think we were using a single writer in the PIP 192 design. I
>>> thought we had many producers sending events to a compacted topic. My
>>> proposal would still have many producers, but the writer to bookkeeper
>>> would act as the single writer. It would technically be distinct from
>>> a normal Pulsar topic producer.
>>>
>>> I should highlight that I am only proposing "broker filtering before
>>> write" in the context of PIP 192 and as an alternative to adding
>>> pluggable compaction strategies. It would not be a generic feature.
>>>
>>>
>> I was worried about the worst case where two producers(leaders) happen to
>> write the compacted topic (although Pulsar can guarantee "a single writer"
>> or "a single producer" for a topic in normal situations).
>>
>>
>>
>>> > Could we clarify how to handle
>>> > the following(edge cases and failure recovery)?
>>> > 0. Is the un-compacted topic a persistent topic or a non-persistent
>>> topic?
>>>
>>> It is a persistent topic.
>>>
>>> > 1. How does the leader recover state from the two topics?
>>>
>>> A leader would recover state by first consuming the whole compacted
>>> topic and then by consuming from 

Re: [DISCUSS] PIP-215: Configurable Topic Compaction Strategy

[Heesung Sohn]

Hi Machel,

Here are my additional comments regarding your earlier email.

- I updated the PIP title to show that this will impact table view as well.

- PIP-192 : https://github.com/apache/pulsar/issues/16691 shows the general
idea of the states and their actions, and I defined the actual states here
in the PR,
https://github.com/apache/pulsar/pull/18079/files#diff-7f9930a5c7896b411f61901cf38371e23ba69e753f460bf7f520f6f800d8321a.
I
will further clarify the bundle state data validation logic when
introducing `BundleStateCompactionStrategy` class. This PIP is to support
CompactionStrategy in general.

- Agreed with your point that we should merge CompactionStrategy APIs. I
updated the interface proposal in the PIP. I replaced `"isValid",
"isMergeEnabled", and "merge"` apis with "compact" api.


Thanks,
Heesung


On Tue, Nov 1, 2022 at 11:26 AM Heesung Sohn 
wrote:

> Hi,
> Thank you for the great comments.
> Please find my comments inline too.
>
> Regards,
> Heesung
>
> On Mon, Oct 31, 2022 at 10:59 PM Michael Marshall 
> wrote:
>
>> > I think we lose a single linearized view.
>>
>> Which linearized view are we losing, and what is the role of that
>> linearized view? I think I might be missing why it is important. I
>> agree that consumers won't know about each unsuccessful attempted
>> acquisition of a bundle, but that seems like unnecessary information
>> to broadcast to every broker in the cluster.
>>
>>
>
> PIP-192 proposes an assignment, transfer, and split protocol(multi-phase
> state changes), relying on early broadcast across brokers, and all brokers
> react to their clients according to the state change notifications --
> brokers could defer any client lookups for bundle x if an
> assignment/transfer/split is ongoing for x(broadcasted early in the topic).
> One early broadcast example is the one that I discussed above, `When the
> topic broadcast is faster than the concurrent assignment requests.` I think
> the prefilter could delay this early broadcast, as it needs to go through
> the additional single-leader compaction path.
>
> The bundle state recovery process is simpler by a single linearized view.
>
> The single linearized view can be easier to debug bundle states. We can
> more easily track where the assignment requests come from and how it is
> compacted in a single linearized view.
>
>
>
>> > I think the leader requires a write-through cache to compact messages
>> based
>> > on the latest states.
>>
> This brings up an important point that I would like to clarify. If we
>> trust the write ahead log as the source of truth, what happens when a
>> bundle has been validly owned by multiple brokers? As a broker starts
>> and consumes from the compacted topic, how do we prevent it from
>> incorrectly thinking that it owns a bundle for some short time period
>> in the case that the ownership topic hasn't yet been compacted to
>> remove old ownership state?
>>
>
> Since the multi-phase transfer protocol involves the source and
> destination broker's actions, the successful transfer should get the source
> and destination broker to have the (near) latest state. For example, if
> some brokers have old ownership states(network partitioned or delayed),
> they will redirect clients to the source(old) broker. However, by the
> transfer protocol, the source broker should have the latest state, so it
> can redirect the client again to the destination broker.
>
> When a broker restarts, it won't start until its BSC state to the (near)
> latest (til the last known messageId at that time).
>
>
>> > Pulsar guarantees "a single writer".
>>
>> I didn't think we were using a single writer in the PIP 192 design. I
>> thought we had many producers sending events to a compacted topic. My
>> proposal would still have many producers, but the writer to bookkeeper
>> would act as the single writer. It would technically be distinct from
>> a normal Pulsar topic producer.
>>
>> I should highlight that I am only proposing "broker filtering before
>> write" in the context of PIP 192 and as an alternative to adding
>> pluggable compaction strategies. It would not be a generic feature.
>>
>>
> I was worried about the worst case where two producers(leaders) happen to
> write the compacted topic (although Pulsar can guarantee "a single writer"
> or "a single producer" for a topic in normal situations).
>
>
>
>> > Could we clarify how to handle
>> > the following(edge cases and failure recovery)?
>> > 0. Is the un-compacted topic a persistent topic or a non-persistent
>> topic?
>>
>> It is a persistent topic.
>>
>> > 1. How does the leader recover state from the two topics?
>>
>> A leader would recover state by first consuming the whole compacted
>> topic and then by consuming from the current location of a cursor on
>> the first input topic. As stated elsewhere, this introduces latency
>> and could be an issue.
>>
>> > 2. How do we handle the case when the leader fails before writing
>> messages
>> > to the compacted topic
>>

Re: [VOTE] Release Apache Pulsar Helm Chart 3.0.0 based on 3.0.0-candidate-1

[Dave Fisher]

+1 (binding)

I checked the signatures and checksums of the source and convenience binary 
packages.
I did a license check of both and got reasonable results. The NOTICE is fine as 
well.

I reviewed the release notes. Very important highlights, great job!

I notice that on release the index.yaml will need to change to point to the 
proper download url for apache.org. That is expected.

Best Regards,
Dave 


> On Oct 20, 2022, at 10:58 PM, Michael Marshall  wrote:
> 
> Hello Apache Pulsar Community,
> 
> This is a call for the vote to release the Apache Pulsar Helm Chart
> version 3.0.0.
> 
> This is our first official vote to release the helm chart. It follows
> a newly merged release process that is still subject to change
> (https://github.com/apache/pulsar-helm-chart/pull/301).
> 
> I did a major version bump to mitigate the security concerns raised by
> https://github.com/apache/pulsar-helm-chart/issues/294.
> 
> The release candidate is available at:
> https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.0.0-candidate-1/
> 
> pulsar-chart-3.0.0-source.tar.gz - is the "main source release".
> pulsar-3.0.0.tgz - is the binary Helm Chart release.
> 
> Public keys are available at: https://www.apache.org/dist/pulsar/KEYS
> 
> Here is a draft of the 3.0.0 release notes:
> https://github.com/apache/pulsar-helm-chart/pull/322
> 
> For convenience "index.yaml" has been uploaded (though excluded from
> voting), so you can also run the below commands.
> 
> helm repo add apache-pulsar-dist-dev
> https://dist.apache.org/repos/dist/dev/pulsar/helm-chart/3.0.0-candidate-1/
> helm repo update
> helm install pulsar apache-pulsar-dist-dev/pulsar
> 
> pulsar-3.0.0.tgz.prov - is also uploaded for verifying Chart
> Integrity, though it is not strictly required for releasing the
> artifact based on ASF Guidelines.
> 
> You can optionally verify this file using this helm plugin
> https://github.com/technosophos/helm-gpg, or by using helm --verify
> (https://helm.sh/docs/helm/helm_verify/). However, I had trouble with
> the formatting of our KEYS file while trying to get the helm --verify
> feature to work.
> 
> helm gpg verify pulsar-3.0.0.tgz
> 
> The vote will be open for at least 72 hours.
> 
> Only votes from PMC members are binding, but members of the community are
> encouraged to test the release and vote with "(non-binding)".
> 
> For license checks, the .rat-excludes files is included, so you can
> run the following to verify licenses (just update $PATH_TO_RAT to
> point to your binary download of
> https://creadur.apache.org/rat/download_rat.cgi):
> 
> tar -xvf pulsar-chart-3.0.0-source.tar.gz
> cd pulsar-chart-3.0.0
> java -jar $PATH_TO_RAT/apache-rat-0.15/apache-rat-0.15.jar . -E .rat-excludes
> 
> Additionally, tests to verify upgrade scenarios are particularly helpful.
> 
> Please note that the version number excludes the `-candidate-X`
> string, so it's now
> simply 3.0.0. This will allow us to rename the artifact without modifying
> the artifact checksums when we actually release it.
> 
> Thanks,
> Michael

[DISCUSS] Do not grant permission for each partition to reduce unnecessary zk metadata

[樊建业]

Hi All,

Now when we grant permission on a partitioned-topic, pulsar broker would try to 
grant permission on each partition. This would result in generating many 
unnecessary zk metadata and updating zk node repeatedly when granting 
permission. 

This problem is shown in https://github.com/apache/pulsar/issues/16768 
 and discussed in 
https://github.com/apache/pulsar/pull/16792 
 .  Actually in the current 
permission module, it is no need to grant permission on each partition when 
creating a multi-partition topic. And in order to meet the compatibility with 
previous pulsar version, there are two implementation. 


Selected implementation
 I prefer to this implementation, described in  
https://github.com/apache/pulsar/pull/18222 
 .
In this implementation, we only change the behaviour of grant permission on 
based topic.


Alternative implementation
Described in https://github.com/apache/pulsar/pull/16792 
 
In this implementation, we would change the behavior of grant permission on 
both based topic and partition. 

Thanks!
Ken

Re: Releasing current master as Pulsar 2.11.0 ?

[Enrico Olivelli]

PengHui,

Il giorno mar 1 nov 2022 alle ore 07:51 PengHui Li
 ha scritto:
>
> > As it is, we already need to discuss EOL for 2.7 and 2.8.
>
> Agree. We should clarify this one.
> I think we can stop to provide new releases for 2.7
> and only security or critical bugs for 2.8 (one more official release)
>
> https://github.com/apache/pulsar/issues/15966 will make the
> release strategy clear.
>
> LTS -> 36 months (24 + 12)
> Feature release -> 6 months (3+3)
>
> Thanks,
> Penghui
>
> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall 
> wrote:
>
> > I am concerned that we have too many active release branches, and planning
> > to follow 2.11.0 with 3.0.0 soon after feels like it will make that problem
> > worse. As it is, we already need to discuss EOL for 2.7 and 2.8.
> >
> > Thanks,
> > Michael
> >
> > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li  wrote:
> >
> > > Releasing from the master branch will bring more uncertainty, no?
> > > We have fixed many regressions that were introduced to branch-2.11.
> > > If we cut a new branch-2.11 based on the master branch. Maybe new
> > > regressions
> > > will happen again. This may make us wait another month to have a 2.11.0
> > > release.

I am not sure.
I don't know if anyone is actively testing the 2.11 branch more than
the master branch.
On my side the (automated) testing that I do with my colleagues on
branch-2.11 is basically the same as for the master branch.

I believe that if we want to cut a 2.11 release that is not branched
again from the master branch
we really must start the release as soon as BK 4.15.3 is released

Many people contributed features to the master branch that cannot be
shipped with 2.11 because
they are considered "breaking changes".
But 2.11 was supposed to be released in August, more than 3 months ago.


Enrico


> > >
> > > IMO, we can start Pulsar 3.0 (follow
> > > https://github.com/apache/pulsar/issues/15966)
> > > after 2.11.0 is released instead of waiting for 3 more months.
> > >
> > > For https://github.com/apache/bookkeeper/issues/3466
> > > I don't think it's a blocker for the Pulsar release for now.
> > > Yes, it is worth investigating more. We also tried a chaos test for that
> > > case.
> > > We haven't reproduced the problem on Pulsar.
> > >
> > > Now, we are just waiting for the new BookKeeper release 4.15.3 since
> > 4.15.2
> > > has regressions [1]
> > >
> > > [1] https://github.com/apache/bookkeeper/pull/3523
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Tue, Nov 1, 2022 at 3:10 AM Michael Marshall 
> > > wrote:
> > >
> > > > I have not followed the branch-2.11 work closely, but I think it makes
> > > > sense to re-create branch-2.11 from the current master.
> > > >
> > > > We created branch-2.11 almost 3 months ago. Re-creating the branch
> > > > will prevent unnecessary delay on new features added over the past 3
> > > > months.
> > > >
> > > > If we follow through with this proposal, we will need to clean up PR
> > > > tags and milestones to prevent confusion.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > On Mon, Oct 31, 2022 at 3:31 AM Enrico Olivelli 
> > > > wrote:
> > > > >
> > > > > Hello Pulsar fellows,
> > > > >
> > > > > I think that too much time passed since we wanted to cut 2.11.
> > > > >
> > > > > The branch-2.11 contains some code used by no one.
> > > > >
> > > > > In the meantime many features went into master branch,
> > > > >
> > > > > I don't think that it is worth it to cut a release from branch-2.11
> > > > > and start with something that is already stale.
> > > > >
> > > > > I propose to drop branch-2.11 and create a new branch out of the
> > > > > current master branch and start the period of hardening before
> > cutting
> > > > > the release.
> > > > >
> > > > > IIUC we are waiting for this BookKeeper issue to be confirmed or
> > fixed
> > > > > or closed as "not a problem":
> > > > > https://github.com/apache/bookkeeper/issues/3466
> > > > > I am personally working on that case together with the folks you
> > > > > created the issue.
> > > > > Honestly I have never been able to reproduce the problem with Pulsar.
> > > > > I believe that it will take at least another week before having more
> > > > > results about the investigations I am doing on BK. The problem is
> > > > > reproducible only on a long-running test (more than 4 hours) of a
> > > > > third party project and only in some private QA environment.
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > Enrico
> > > >
> > >
> >
>
> On Tue, Nov 1, 2022 at 2:15 PM Michael Marshall 
> wrote:
>
> > I am concerned that we have too many active release branches, and planning
> > to follow 2.11.0 with 3.0.0 soon after feels like it will make that problem
> > worse. As it is, we already need to discuss EOL for 2.7 and 2.8.
> >
> > Thanks,
> > Michael
> >
> > On Mon, Oct 31, 2022 at 7:55 PM PengHui Li  wrote:
> >
> > > Releasing from the master branch will bring more uncertainty, no?
> > > We have fixed many regressions that were 

[DISCUSS] PIP-222: Add CommandPartitionedTopicMetadata metrics

[Jiuming Tao]

Hi pulsar community,

I’ve opened a PIP to discuss: PIP-222: Add CommandPartitionedTopicMetadata 
metrics

The PIP link: https://github.com/apache/pulsar/issues/18319 


Thanks,
Tao Jiuming

Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive

[丛搏]

Good question!
It is a scene for our subsequent optimization, and we are implementing
it with minimal changes.

Asaf Mesika  于2022年11月3日周四 16:49写道：
>
> Regarding the implementation suggested in the PIP:
>
> It said that you will read the messages from all partitions assigned, and
> once you see a message from a different partition you halt and supply the
> batch.
> My question is: Why can't we simply read the messages from one partition
> (say 10 messages) and then the next read from the other partition?
The amount of data in a partition is not large, and there are many
partitions. In this scenario, this method may have a large delay.
>
> On Thu, Oct 27, 2022 at 3:36 PM Haiting Jiang 
> wrote:
>
> > +1
> >
> > Haiting
> >
> > On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
> >  wrote:
> > >
> > > +1
> > >
> > > On Tue, Oct 25, 2022 at 12:25 PM 丛搏  wrote:
> > >
> > > > Hi, pulsar community:
> > > >
> > > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > > partitioned topic or multi-topics`
> > > >
> > > > discuss thread:
> > > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > >
> > > > PIP: https://github.com/apache/pulsar/issues/18182
> > > >
> > > > Thanks,
> > > > bo
> > > >
> >

[GitHub] [pulsar] liuhuagui added a comment to the discussion: Does pulsar delay message support rebooking?

[GitBox]

GitHub user liuhuagui added a comment to the discussion: Does pulsar delay 
message support rebooking?

Thanks！

GitHub link: 
https://github.com/apache/pulsar/discussions/18317#discussioncomment-4045310


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] coderzc added a comment to the discussion: Does pulsar delay message support rebooking?

[GitBox]

GitHub user coderzc added a comment to the discussion: Does pulsar delay 
message support rebooking?

Pulsar does not support changing the scheduled time of the message that has 
been sent. No plan for the time being.

GitHub link: 
https://github.com/apache/pulsar/discussions/18317#discussioncomment-4045149


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] tisonkun added a comment to the discussion: Publish On official page

[GitBox]

GitHub user tisonkun added a comment to the discussion: Publish On official page

@vinayppsharma I answered the corresponding email. And for this specific 
question, you may refer to 
https://pulsar.apache.org/contribute/documentation/contribution

GitHub link: 
https://github.com/apache/pulsar/discussions/18303#discussioncomment-4045036


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive

[Asaf Mesika]

Regarding the implementation suggested in the PIP:

It said that you will read the messages from all partitions assigned, and
once you see a message from a different partition you halt and supply the
batch.
My question is: Why can't we simply read the messages from one partition
(say 10 messages) and then the next read from the other partition?

On Thu, Oct 27, 2022 at 3:36 PM Haiting Jiang 
wrote:

> +1
>
> Haiting
>
> On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
>  wrote:
> >
> > +1
> >
> > On Tue, Oct 25, 2022 at 12:25 PM 丛搏  wrote:
> >
> > > Hi, pulsar community:
> > >
> > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > partitioned topic or multi-topics`
> > >
> > > discuss thread:
> > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > >
> > > PIP: https://github.com/apache/pulsar/issues/18182
> > >
> > > Thanks,
> > > bo
> > >
>

[GitHub] [pulsar] liuhuagui edited a discussion: Does pulsar delay message support rebooking?

[GitBox]

GitHub user liuhuagui edited a discussion: Does pulsar delay message support 
rebooking?

 For example, the previous delay time was changed from 5m to 30m or even 0, 
even deleted.

 If not, is there a plan?

GitHub link: https://github.com/apache/pulsar/discussions/18317


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] liuhuagui added a comment to the discussion: Does pulsar delay message support rebooking?

[GitBox]

GitHub user liuhuagui added a comment to the discussion: Does pulsar delay 
message support rebooking?

@coderzc Please help to answer

GitHub link: 
https://github.com/apache/pulsar/discussions/18317#discussioncomment-4044813


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: [VOTE] PIP-218: Consumer batchReceive() single partition every receive

[丛搏]

Hi all:
Thank you all for voting.
I am closing the vote with 3 binding +1 votes (PengHui, Enrico,
Haiting) and 1 non-binding +1 vote (Yunze).

The implementation pr is :
https://github.com/apache/pulsar/pull/18316

Thanks,
Bo

Enrico Olivelli  于2022年10月28日周五 17:58写道：
>
> +1 (binding)
>
> Enrico
>
> Il giorno ven 28 ott 2022 alle ore 09:48 PengHui Li
>  ha scritto:
> >
> > +1 (binding)
> >
> > Thanks,
> > Penghui
> >
> > On Fri, Oct 28, 2022 at 1:32 PM Haiting Jiang 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Haiting
> > >
> > > On Fri, Oct 28, 2022 at 10:43 AM Yunze Xu 
> > > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Thu, Oct 27, 2022 at 9:14 PM 丛搏  wrote:
> > > > >
> > > > > Hi, pulsar community:
> > > > >
> > > > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > > > partitioned topic or multi-topics`
> > > > >
> > > > > discuss thread:
> > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > > >
> > > > > PIP: https://github.com/apache/pulsar/issues/18182
> > > > >
> > > > > Thanks,
> > > > > bo
> > >

Re: [DISCUSS] Release Pulsar 2.9.4

[PengHui Li]

+1

Thanks,
Penghui

> On Nov 3, 2022, at 14:01, 丛搏  wrote:
> 
> Hello, Pulsar community:
> 
> I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
> three months since 2.9.3 was released.
> 
> There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
> cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
> directly into branch-2.9.
> 
> There are 33 PRs [1] opened. I'll follow up on each of those PRs to
> see if they will be completed soon or will need to be pushed to 2.9.5
> 
> If you have any important fixes or any questions, please reply to this
> email, and we will evaluate whether to include them in 2.9.4
> 
> Thanks,
> Bo
> [0] - 
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
> [1] - 
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
> I will

[DISCUSS] Release Pulsar 2.9.4

[丛搏]

Hello, Pulsar community:

I'd like to propose releasing Apache Pulsar 2.9.4. It's been about
three months since 2.9.3 was released.

There are 123 PRs [0] needed to cherry-pick in branch-2.9. I will
cherry-pick these PRs for branch-2.9. Exclude some PRs that merge
directly into branch-2.9.

There are 33 PRs [1] opened. I'll follow up on each of those PRs to
see if they will be completed soon or will need to be pushed to 2.9.5

If you have any important fixes or any questions, please reply to this
email, and we will evaluate whether to include them in 2.9.4

Thanks,
Bo
[0] - 
https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+-label%3Acherry-picked%2Fbranch-2.9+is%3Amerged
[1] - 
https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.9.4+is%3Aopen
I will