Re: [PROPOSAL] Roadmap for 3.0 release

[r...@apache.org]

+1

--
Thanks
Xiaolong Ran

Yubiao Feng  于2023年3月3日周五 14:38写道：

> The same as Zike (^_^)
>
> Thanks to Matteo for initiating this release roadmap. I am interested in
> this release. I'd also like to volunteer as a release manager for Apache
> Pulsar 3.0.0
>
> Thanks
> Yubiao Feng
>
> On Sat, Feb 18, 2023 at 6:44 AM Matteo Merli  wrote:
>
> > Since the LTS release model has been formally approved, I'm proposing
> > the following schedule for the release:
> >
> >  * Tue - 2023-05-11
> >- RC-1
> >- Code Freeze -- Only critical fixes will be merged in the 3.0
> > release branch. Contributors should plan to have all the changes merged
> in
> > before this date. Exceptions should be extremely rare and strongly
> > motivated.
> >
> >  * Tue - 2023-05-18 - RC-2
> >  * Tue - 2023-05-25 - RC-3
> >  * Tue - 2023-05-02 - Announce 3.0 Release
> >
> > These dates will be published on the website to present users with a
> > "roadmap" and we should commit to and respect these dates.
> >
> > I also wanted to propose trying out a model where we have 3 release
> > managers for all major releases.
> >
> > The reasoning behind this is for this small group of people to
> collaborate
> > and divide the tasks for the release: merging patches from the "master"
> > branch, preparing RC, and testing.
> >
> > Since everyone also has other work duties and unexpected tasks that can
> pop
> > up at any time, it will help to have redundancy in the release-management
> > "team", so that we can release on the exact dates.
> >
> > Thanks,
> > Matteo
> >
> > --
> > Matteo Merli
> > 
> >
>

Re: [PROPOSAL] Roadmap for 3.0 release

[Yubiao Feng]

The same as Zike (^_^)

Thanks to Matteo for initiating this release roadmap. I am interested in
this release. I'd also like to volunteer as a release manager for Apache
Pulsar 3.0.0

Thanks
Yubiao Feng

On Sat, Feb 18, 2023 at 6:44 AM Matteo Merli  wrote:

> Since the LTS release model has been formally approved, I'm proposing
> the following schedule for the release:
>
>  * Tue - 2023-05-11
>- RC-1
>- Code Freeze -- Only critical fixes will be merged in the 3.0
> release branch. Contributors should plan to have all the changes merged in
> before this date. Exceptions should be extremely rare and strongly
> motivated.
>
>  * Tue - 2023-05-18 - RC-2
>  * Tue - 2023-05-25 - RC-3
>  * Tue - 2023-05-02 - Announce 3.0 Release
>
> These dates will be published on the website to present users with a
> "roadmap" and we should commit to and respect these dates.
>
> I also wanted to propose trying out a model where we have 3 release
> managers for all major releases.
>
> The reasoning behind this is for this small group of people to collaborate
> and divide the tasks for the release: merging patches from the "master"
> branch, preparing RC, and testing.
>
> Since everyone also has other work duties and unexpected tasks that can pop
> up at any time, it will help to have redundancy in the release-management
> "team", so that we can release on the exact dates.
>
> Thanks,
> Matteo
>
> --
> Matteo Merli
> 
>

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

Hi, Eric. Thank you very much for the work you have done. I have no
objections to the process, and it would be even better if there could be a
rough timeline.

Thanks
ZhangJian He


On Fri, 3 Mar 2023 at 09:06, Shen Eric  wrote:

> Hi Zhangjian,
>
> I am a PM from StreamNative and we also had some internal discussions
> related to this topic. Let me share our ongoing planning:
>
>   *   We will extract the pulsar admin pkg from the pulsarctl to a
> separate open repo which will be called pulsar-admin-go under StreamNative.
>   *   Will iterate the pulsar-admin-go library by adding more tests,
> documentation and may also update or fix the existing APIs.
>   *   After the pulsar-admin-go library is stable, we will contribute this
> project to Apache Foundation.
>
> Do you think this plan works for you?
>
> On 2023/02/17 01:47:26 ZhangJian He wrote:
> > I would like to express that the current Pulsar client for Go
> > (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> > to propose that we work towards adding this feature to pulsar-client-go.
> >
> > I believe that this new feature would be a valuable addition to
> > pulsar-client-go, and I am excited to work to make it happen.
> >
> > I have submitted a PR:
> https://github.com/apache/pulsar-client-go/pull/959
> > The full api is not currently available, but we are adding.
> >
> > Below is a simple example about how to use
> >
> > ## usage
> >
> > ```go
> > package main
> >
> > import (
> > "fmt"
> > "github.com/apache/pulsar-client-go/padmin"
> > )
> >
> > func main() {
> > admin, err := padmin.NewDefaultPulsarAdmin()
> > if err != nil {
> > panic(err)
> > }
> > // get namespace topic list
> > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > "namespace")
> > if err != nil {
> > panic(err)
> > }
> > fmt.Println(topics)
> > }
> > ```
> >
> > Thanks
> > ZhangJian He
> >
>
>
> --
>
> Best regards,
>
> Eric Shen 沈瑀昊
>

Re: [PROPOSAL] Roadmap for 3.0 release

[Zike Yang]

Hi,

Thanks to Matteo for initiating this release roadmap. I am interested
in this release.

I'd also like to volunteer as a release manager for Apache Pulsar 3.0.0.

Thanks,
Zike Yang

On Tue, Feb 21, 2023 at 10:56 PM tison  wrote:
>
> +1
>
> Thanks for driving this effort!
>
> Also, I created https://github.com/apache/pulsar/issues/19570 to
> correspondingly update our website. Feel free to leave a comment. I'm going
> to send a draft in days.
>
> Best,
> tison.
>
>
> Enrico Olivelli  于2023年2月21日周二 19:20写道：
>
> > +1
> >
> > Perfect
> >
> > Enrico
> >
> > Il giorno mar 21 feb 2023 alle ore 08:11 Haiting Jiang
> >  ha scritto:
> > >
> > > +1 Looks great!
> > >
> > >
> > > Haiting
> > >
> > > On Tue, Feb 21, 2023 at 12:11 PM Hang Chen  wrote:
> > > >
> > > > +1
> > > >
> > > > Thanks,
> > > > Hang
> > > >
> > > > Christophe Bornet  于2023年2月20日周一 21:46写道：
> > > > >
> > > > > +1
> > > > >
> > > > > Also, I'd like to volunteer as a release manager for this release.
> > > > >
> > > > > Christophe
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Le ven. 17 févr. 2023 à 23:44, Matteo Merli  a
> > écrit :
> > > > > >
> > > > > > Since the LTS release model has been formally approved, I'm
> > proposing
> > > > > > the following schedule for the release:
> > > > > >
> > > > > >  * Tue - 2023-05-11
> > > > > >- RC-1
> > > > > >- Code Freeze -- Only critical fixes will be merged in the
> > 3.0
> > > > > > release branch. Contributors should plan to have all the changes
> > merged in
> > > > > > before this date. Exceptions should be extremely rare and strongly
> > > > > > motivated.
> > > > > >
> > > > > >  * Tue - 2023-05-18 - RC-2
> > > > > >  * Tue - 2023-05-25 - RC-3
> > > > > >  * Tue - 2023-05-02 - Announce 3.0 Release
> > > > > >
> > > > > > These dates will be published on the website to present users with
> > a
> > > > > > "roadmap" and we should commit to and respect these dates.
> > > > > >
> > > > > > I also wanted to propose trying out a model where we have 3 release
> > > > > > managers for all major releases.
> > > > > >
> > > > > > The reasoning behind this is for this small group of people to
> > collaborate
> > > > > > and divide the tasks for the release: merging patches from the
> > "master"
> > > > > > branch, preparing RC, and testing.
> > > > > >
> > > > > > Since everyone also has other work duties and unexpected tasks
> > that can pop
> > > > > > up at any time, it will help to have redundancy in the
> > release-management
> > > > > > "team", so that we can release on the exact dates.
> > > > > >
> > > > > > Thanks,
> > > > > > Matteo
> > > > > >
> > > > > > --
> > > > > > Matteo Merli
> > > > > > 
> >

Re: [Discussion] Allowing configure if function consumer should skip to latest

[PengHui Li]

Hi Neng,

Thanks for raising up the discussion

> In certain failure cases, the function needs to skip all the content
between the last successfully acked message and the latest message in the
topic in order to skip the huge backlog and quick recovery.

Do you have some real cases that can help us to understand it
is necessary to introduce a new flag? Another possibility is users
can use pulsar admin to reset the cursor to the latest position,
Why will it not work for users? 

Regards,
Penghui

> On Mar 1, 2023, at 10:16, Neng Lu  wrote:
> 
> In certain failure cases, the function needs to skip all the content
> between the last successfully acked message and the latest message in the
> topic in order to skip the huge backlog and quick recovery.

Re: [DISCUSS] PMC/Committer Emiratus status

[Yunze Xu]

As a PMC member, I don't like playing a game of determining who should
be removed from PMC as well.

I hear a viewpoint that someone is only participating in the community
only to join a PMC so that he can benefit from it. After becoming a
PMC member, he is never active in the community. It might be true but
I think it's acceptable. Making such a rule won't prevent such cases.
If he wants, he can make use of the rule and keep himself "active" to
avoid being kicked out of the PMC. Though the active state is fake.

I'm not against the way to remove (or something else that sounds good)
a PMC member because none of these ways is perfect. However, I'm
STRONGLY AGAINST changing a rule that has been applied for some time
unless it can be proved the rule is very harmful to the community.
You mentioned https://www.apache.org/dev/pmc.html#pmc-removal. But
please don't ignore the first sentence:

> Projects can establish their own policy on handling inactive members, as long 
> as they apply it CONSISTENTLY.

In addition, Dave and Tison both mentioned we have some boards or
webpages to see how many people are active. We don't need to remove
some PMC members just for knowing who were still active recently.

BTW, I'm also curious about the motivation of this proposal. I'm
wondering how do the inactive PMC members harm the community?

Thanks,
Yunze

On Fri, Mar 3, 2023 at 10:14 AM tison  wrote:
>
> Hi,
>
> In the proposal, it's unclear if you'd like to _mark_ the inactive members
> in emeritus status or _remove_ them from the LDAP group.
>
> I saw a similar discussion in the Flink community, resulting in "active"
> sentences in its Bylaws[1]. Here is some consensus there:
>
> 1. Merits never expire. There's no reason to _remove_ a committer or PMC
> member from the LDAP group because of inactive following the Apache way. I
> remember numbered cases a member got removed because they _keep_ harming
> the community.
> 2. Emeritus status is set for unblocking consensus. The Flink community
> experienced some votes that could not get the required approvals in time
> and thus tried to unblock consensus by setting some members with binding
> votes in emeritus status. Do we spot concrete issues that the Pulsar
> community cannot work well with current PMC members and committers group?
> 3. Emeritus status is voluntary. I know that in other foundations, it can
> be judged or eagerly applied, but in ASF, we share a "Community of Peers"
> sense that everyone is a volunteer. They won't be "fired" because of "low
> productivity".
>
> > Gain real visibility into the health of the project in terms of real
> active PMC / Committers members.
>
> On the community page, we already have a monthly active contributor graph.
> It's an insight concept; I don't think we should _remove_ members for such
> a reason.
>
> > Have the alert threshold set correctly as to when it's time to start
> working on recruiting new PMC / Committers members
>
> Ditto. When working in the community, you will know what modules or repos
> lack participants. For example, I remember someone proposing to promote
> more committers working on Pulsar multilingual clients. It's not a reason
> for emeritus or removal.
>
> ---
>
> Generally, committers and PMC members have "Earned Authority" to be in the
> group. They share a high trust level, and I have numerous examples that
> returned members do outstanding work. If we don't have some critical issues
> to introduce an emeritus status, and such members do no harm, why do we set
> a bar if they return?
>
> Best,
> tison.
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
>
>
> Dave Fisher  于2023年3月3日周五 09:37写道：
>
> > Hi,
> >
> > > On Mar 2, 2023, at 11:58 AM, Asaf Mesika  wrote:
> > >
> > > Hi,
> > >
> > > Following the discussion I've started in Pulsar bi-weekly meetings.
> >
> > As I said when you brought this up, I don’t think it is a good idea, not a
> > good idea at all.
> >
> > > Projects can establish their own policy on handling inactive members, as
> > > long as they apply it consistently.
> >
> >
> > As a PMC member I have no desire to play a game of consistently tossing
> > PMC members who somehow haven’t met an engagement criteria. That is an
> > anti-pattern to building a community. It would be disruptive and time
> > consuming.
> >
> > >
> > > = The idea
> > > PMC and Committers members will transition into Emeritus status after X
> > > months of inactivity, or voluntarily.
> >
> > PMC members have the opportunity with or without this proposal to
> > voluntarily resign as PMC members with or without giving up their commit
> > bit.
> >
> > >
> > > = Why?
> > > - Gain real visibility into the health of the project in terms of real
> > > active PMC / Committers members.
> >
> > Here is an exercise. Generate activity reports to show criteria. We can
> > correlate between the rosters and all of our GitHub repositories. Also
> > Mailing lists and slack which only goes back 90 days. I would be 

Re: [DISCUSS] PMC/Committer Emiratus status

[tison]

Hi,

In the proposal, it's unclear if you'd like to _mark_ the inactive members
in emeritus status or _remove_ them from the LDAP group.

I saw a similar discussion in the Flink community, resulting in "active"
sentences in its Bylaws[1]. Here is some consensus there:

1. Merits never expire. There's no reason to _remove_ a committer or PMC
member from the LDAP group because of inactive following the Apache way. I
remember numbered cases a member got removed because they _keep_ harming
the community.
2. Emeritus status is set for unblocking consensus. The Flink community
experienced some votes that could not get the required approvals in time
and thus tried to unblock consensus by setting some members with binding
votes in emeritus status. Do we spot concrete issues that the Pulsar
community cannot work well with current PMC members and committers group?
3. Emeritus status is voluntary. I know that in other foundations, it can
be judged or eagerly applied, but in ASF, we share a "Community of Peers"
sense that everyone is a volunteer. They won't be "fired" because of "low
productivity".

> Gain real visibility into the health of the project in terms of real
active PMC / Committers members.

On the community page, we already have a monthly active contributor graph.
It's an insight concept; I don't think we should _remove_ members for such
a reason.

> Have the alert threshold set correctly as to when it's time to start
working on recruiting new PMC / Committers members

Ditto. When working in the community, you will know what modules or repos
lack participants. For example, I remember someone proposing to promote
more committers working on Pulsar multilingual clients. It's not a reason
for emeritus or removal.

---

Generally, committers and PMC members have "Earned Authority" to be in the
group. They share a high trust level, and I have numerous examples that
returned members do outstanding work. If we don't have some critical issues
to introduce an emeritus status, and such members do no harm, why do we set
a bar if they return?

Best,
tison.

[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws


Dave Fisher  于2023年3月3日周五 09:37写道：

> Hi,
>
> > On Mar 2, 2023, at 11:58 AM, Asaf Mesika  wrote:
> >
> > Hi,
> >
> > Following the discussion I've started in Pulsar bi-weekly meetings.
>
> As I said when you brought this up, I don’t think it is a good idea, not a
> good idea at all.
>
> > Projects can establish their own policy on handling inactive members, as
> > long as they apply it consistently.
>
>
> As a PMC member I have no desire to play a game of consistently tossing
> PMC members who somehow haven’t met an engagement criteria. That is an
> anti-pattern to building a community. It would be disruptive and time
> consuming.
>
> >
> > = The idea
> > PMC and Committers members will transition into Emeritus status after X
> > months of inactivity, or voluntarily.
>
> PMC members have the opportunity with or without this proposal to
> voluntarily resign as PMC members with or without giving up their commit
> bit.
>
> >
> > = Why?
> > - Gain real visibility into the health of the project in terms of real
> > active PMC / Committers members.
>
> Here is an exercise. Generate activity reports to show criteria. We can
> correlate between the rosters and all of our GitHub repositories. Also
> Mailing lists and slack which only goes back 90 days. I would be better
> persuaded if you did that to actually show and prove that there is a
> problem. I think you will find that it is a large amount of effort with
> little value in the end.
>
> > - Have the alert threshold set correctly as to when it's time to start
> > working on recruiting new PMC / Committers members.
>
> Those two points are not coupled. We ARE ALWAYS be on the alert for new
> committers and PMC Members. This PMC has been ACTIVE in recognizing many of
> those who are deserving.
>
> Here are the Pulsar Board reports:
> https://whimsy.apache.org/board/minutes/Pulsar
>
> >
> > = Is there any precedence?
> > Yes. A lot.
> > Many CNCF projects do it.
> > Many Apache projects do it.
>
> I’ve been on many PMC’s and I cannot think of one that does this. You’ve
> come up with a few examples below, but I won’t be persuaded.
>
> > Apache foundations rules allow it.
> >
> > Read below to see examples and links.
> >
> >
> > = Examples
> >
> > === etcD project <
> https://github.com/etcd-io/etcd/blob/main/GOVERNANCE.md>
>
> I don’t care about how another Foundation does it.
>
> >
> > Quote
> >
> > Life priorities, interests, and passions can change. Maintainers can
> retire
> > and move to the emeritus status
> > <
> https://github.com/etcd-io/etcd/blob/main/README.md#etcd-emeritus-maintainers
> >.
> > If a maintainer needs to step down, they should inform other maintainers,
> > if possible, help find someone to pick up the related work. At the very
> > least, ensure the related work can be continued. Afterward they can
> remove
> > themselves from 

Re: [DISCUSS] PMC/Committer Emiratus status

[Dave Fisher]

Hi,

> On Mar 2, 2023, at 11:58 AM, Asaf Mesika  wrote:
> 
> Hi,
> 
> Following the discussion I've started in Pulsar bi-weekly meetings.

As I said when you brought this up, I don’t think it is a good idea, not a good 
idea at all.

> Projects can establish their own policy on handling inactive members, as
> long as they apply it consistently.


As a PMC member I have no desire to play a game of consistently tossing PMC 
members who somehow haven’t met an engagement criteria. That is an anti-pattern 
to building a community. It would be disruptive and time consuming.

> 
> = The idea
> PMC and Committers members will transition into Emeritus status after X
> months of inactivity, or voluntarily.

PMC members have the opportunity with or without this proposal to voluntarily 
resign as PMC members with or without giving up their commit bit.

> 
> = Why?
> - Gain real visibility into the health of the project in terms of real
> active PMC / Committers members.

Here is an exercise. Generate activity reports to show criteria. We can 
correlate between the rosters and all of our GitHub repositories. Also Mailing 
lists and slack which only goes back 90 days. I would be better persuaded if 
you did that to actually show and prove that there is a problem. I think you 
will find that it is a large amount of effort with little value in the end.

> - Have the alert threshold set correctly as to when it's time to start
> working on recruiting new PMC / Committers members.

Those two points are not coupled. We ARE ALWAYS be on the alert for new 
committers and PMC Members. This PMC has been ACTIVE in recognizing many of 
those who are deserving.

Here are the Pulsar Board reports: 
https://whimsy.apache.org/board/minutes/Pulsar

> 
> = Is there any precedence?
> Yes. A lot.
> Many CNCF projects do it.
> Many Apache projects do it.

I’ve been on many PMC’s and I cannot think of one that does this. You’ve come 
up with a few examples below, but I won’t be persuaded.

> Apache foundations rules allow it.
> 
> Read below to see examples and links.
> 
> 
> = Examples
> 
> === etcD project 

I don’t care about how another Foundation does it.

> 
> Quote
> 
> Life priorities, interests, and passions can change. Maintainers can retire
> and move to the emeritus status
> .
> If a maintainer needs to step down, they should inform other maintainers,
> if possible, help find someone to pick up the related work. At the very
> least, ensure the related work can be continued. Afterward they can remove
> themselves from list of existing maintainers.
> 
> If a maintainer is not been performing their duties for period of 12
> months, they can be removed by other maintainers. In that case inactive
> maintainer will be first notified via an email. If situation doesn't
> improve, they will be removed. If an emeritus maintainer wants to regain an
> active role, they can do so by renewing their contributions. Active
> maintainers should welcome such a move. Retiring of other maintainers or
> regaining the status should require approval of at least two active
> maintainers.
> 
> === Apache Gump
> According to this link , they have
> emeritus status for maintainers and PMC members and policy to transition.

Gump is a tiny project that does not release code. The two PMC members was 
added in 2014. The rest were in 2004 - 2006. The project is a build system that 
other projects used to use. I don’t even know if any project still uses it. In 
fact that are just keeping it up for Tomcat. See 
https://whimsy.apache.org/board/minutes/Gump

> 
> QUOTE
> Committer access is by invitation only and must be approved by lazy
> consensus of the active PMC members. A Committer is considered emeritus by
> their own declaration or by not contributing in any form to the project for
> over six months. An emeritus committer may request reinstatement of commit
> access from the PMC. Such reinstatement is subject to lazy consensus of
> active PMC members.

All of their committers except for on of 16 are ASF Members.

> 
> Membership of the PMC is by invitation only and must be approved by a lazy
> consensus of active PMC members. A PMC member is considered "emeritus" by
> their own declaration or by not contributing in any form to the project for
> over six months. An emeritus member may request reinstatement to the PMC.
> Such reinstatement is subject to lazy consensus of the active PMC members.
> Membership of the PMC can be revoked by an unanimous vote of all the active
> PMC members other than the member in question.
> 
> END QUOTE
> 
> There are many more: Apache Hive
> ,

Hive has 56 PMC Members and 106 Committers. I can tell from their roster that 
many do not seem to be engaged any more.

Board reports: 

RE: Introducer Pulsar admin api to pulsar-client-go

[Shen Eric]

Hi Zhangjian,

I am a PM from StreamNative and we also had some internal discussions related 
to this topic. Let me share our ongoing planning:

  *   We will extract the pulsar admin pkg from the pulsarctl to a separate 
open repo which will be called pulsar-admin-go under StreamNative.
  *   Will iterate the pulsar-admin-go library by adding more tests, 
documentation and may also update or fix the existing APIs.
  *   After the pulsar-admin-go library is stable, we will contribute this 
project to Apache Foundation.

Do you think this plan works for you?

On 2023/02/17 01:47:26 ZhangJian He wrote:
> I would like to express that the current Pulsar client for Go
> (pulsar-client-go) is missing the pulsar Admin API. As such, I would like
> to propose that we work towards adding this feature to pulsar-client-go.
>
> I believe that this new feature would be a valuable addition to
> pulsar-client-go, and I am excited to work to make it happen.
>
> I have submitted a PR: https://github.com/apache/pulsar-client-go/pull/959
> The full api is not currently available, but we are adding.
>
> Below is a simple example about how to use
>
> ## usage
>
> ```go
> package main
>
> import (
> "fmt"
> "github.com/apache/pulsar-client-go/padmin"
> )
>
> func main() {
> admin, err := padmin.NewDefaultPulsarAdmin()
> if err != nil {
> panic(err)
> }
> // get namespace topic list
> topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> "namespace")
> if err != nil {
> panic(err)
> }
> fmt.Println(topics)
> }
> ```
>
> Thanks
> ZhangJian He
>


--

Best regards,

Eric Shen 沈�r昊

[DISCUSS] PIP-253: Expose producer metrics for deadLetterProducer and retryLetterProducer

[Kai Levy]

Hello!

I created a new PIP because I discovered there's no way for a user to
access the metrics for a consumer's deadLetterProducer /
retryLetterProducer, since it is private to ConsumerImpl.java. I would like
to propose an API change that would expose those statistics. More details
on the github issue:
https://github.com/apache/pulsar/issues/19698

Thanks!
Kai

Re: [DISCUSS] PMC/Committer Emiratus status

[P. Taylor Goetz]

I would proceed carefully with this, and would recommend against removing PMC 
members who are “inactive”. Rather, I would recommend a project-level 
“active/inactive” flag that PMC members can voluntarily apply to themselves. 
For example, do a PMC roll call on private@pulsar.a.o 
 and ask whether current PMC members self describe 
as “active” or “inactive”. That status could then be reflected on the Community 
section of the pulsar website. No reply from a PMC member? Mark them inactive 
until they request a change.

Yes, some Apache projects do purge PMCs of “inactive” members, which in my 
opinion is a mistake. Another mantra offered by the Apache Way is that merit 
never expires. 

For example, in my case I’m not particularly active in terms of code 
contributions, etc., but as a former mentor to the project I keep an eye on the 
mailing lists to make sure any potential problems get addressed, etc. I would 
consider myself “active” even though I’m relatively quiet.

So to summarize:

1. You can maintain active/inactive status at the project level with a simple 
flag on a community page, without removing people from the PMC.
2. By making it an informal, self-reported flag, you avoid the overhead of 
board resolutions, etc. and just manage it at the community level. If someone 
wants to change their status, they can just say so or submit a pull request to 
change their status on the pulsar website.
3. Merit doesn’t expire.
4. Purging inactive PMC members unnecessarily adds work for the PMC, Chair,  
and ASF board.

Finally, there’s nothing wrong with having a large PMC even if a significant 
number are “inactive”. I’ve recently witnessed on ASF project avoid the attic 
because several PMC members who had been quiet for years showed up and voted. 
Had that project purged those “inactive” PMC members it would have potentially 
died in the Attic as a result.

Just my $.02. Ultimately the decision is up to the broader Pulsar community to 
make a decision.

-Taylor



> On Mar 2, 2023, at 2:58 PM, Asaf Mesika  wrote:
> 
> Hi,
> 
> Following the discussion I've started in Pulsar bi-weekly meetings.
> 
> = The idea
> PMC and Committers members will transition into Emeritus status after X
> months of inactivity, or voluntarily.
> 
> = Why?
> - Gain real visibility into the health of the project in terms of real
> active PMC / Committers members.
> - Have the alert threshold set correctly as to when it's time to start
> working on recruiting new PMC / Committers members.
> 
> = Is there any precedence?
> Yes. A lot.
> Many CNCF projects do it.
> Many Apache projects do it.
> Apache foundations rules allow it.
> 
> Read below to see examples and links.
> 
> 
> = Examples
> 
> === etcD project 
> 
> Quote
> 
> Life priorities, interests, and passions can change. Maintainers can retire
> and move to the emeritus status
> .
> If a maintainer needs to step down, they should inform other maintainers,
> if possible, help find someone to pick up the related work. At the very
> least, ensure the related work can be continued. Afterward they can remove
> themselves from list of existing maintainers.
> 
> If a maintainer is not been performing their duties for period of 12
> months, they can be removed by other maintainers. In that case inactive
> maintainer will be first notified via an email. If situation doesn't
> improve, they will be removed. If an emeritus maintainer wants to regain an
> active role, they can do so by renewing their contributions. Active
> maintainers should welcome such a move. Retiring of other maintainers or
> regaining the status should require approval of at least two active
> maintainers.
> 
> === Apache Gump
> According to this link , they have
> emeritus status for maintainers and PMC members and policy to transition.
> 
> QUOTE
> Committer access is by invitation only and must be approved by lazy
> consensus of the active PMC members. A Committer is considered emeritus by
> their own declaration or by not contributing in any form to the project for
> over six months. An emeritus committer may request reinstatement of commit
> access from the PMC. Such reinstatement is subject to lazy consensus of
> active PMC members.
> 
> Membership of the PMC is by invitation only and must be approved by a lazy
> consensus of active PMC members. A PMC member is considered "emeritus" by
> their own declaration or by not contributing in any form to the project for
> over six months. An emeritus member may request reinstatement to the PMC.
> Such reinstatement is subject to lazy consensus of the active PMC members.
> Membership of the PMC can be revoked by an unanimous vote of all the active
> PMC members other than the member in question.
> 
> END QUOTE
> 
> There are many more: Apache Hive
> 

[DISCUSS] PMC/Committer Emiratus status

[Asaf Mesika]

Hi,

Following the discussion I've started in Pulsar bi-weekly meetings.

= The idea
PMC and Committers members will transition into Emeritus status after X
months of inactivity, or voluntarily.

= Why?
- Gain real visibility into the health of the project in terms of real
active PMC / Committers members.
- Have the alert threshold set correctly as to when it's time to start
working on recruiting new PMC / Committers members.

= Is there any precedence?
Yes. A lot.
Many CNCF projects do it.
Many Apache projects do it.
Apache foundations rules allow it.

Read below to see examples and links.


= Examples

=== etcD project 

Quote

Life priorities, interests, and passions can change. Maintainers can retire
and move to the emeritus status
.
If a maintainer needs to step down, they should inform other maintainers,
if possible, help find someone to pick up the related work. At the very
least, ensure the related work can be continued. Afterward they can remove
themselves from list of existing maintainers.

If a maintainer is not been performing their duties for period of 12
months, they can be removed by other maintainers. In that case inactive
maintainer will be first notified via an email. If situation doesn't
improve, they will be removed. If an emeritus maintainer wants to regain an
active role, they can do so by renewing their contributions. Active
maintainers should welcome such a move. Retiring of other maintainers or
regaining the status should require approval of at least two active
maintainers.

=== Apache Gump
According to this link , they have
emeritus status for maintainers and PMC members and policy to transition.

QUOTE
Committer access is by invitation only and must be approved by lazy
consensus of the active PMC members. A Committer is considered emeritus by
their own declaration or by not contributing in any form to the project for
over six months. An emeritus committer may request reinstatement of commit
access from the PMC. Such reinstatement is subject to lazy consensus of
active PMC members.

Membership of the PMC is by invitation only and must be approved by a lazy
consensus of active PMC members. A PMC member is considered "emeritus" by
their own declaration or by not contributing in any form to the project for
over six months. An emeritus member may request reinstatement to the PMC.
Such reinstatement is subject to lazy consensus of the active PMC members.
Membership of the PMC can be revoked by an unanimous vote of all the active
PMC members other than the member in question.

END QUOTE

There are many more: Apache Hive
,
Apache Orc ,
...

= What does Apache thinks about this?

According to this link ,
any project can have their policies for retire an inactive PMC member.

QUOTE
SHOULD A PMC REMOVE INACTIVE MEMBERS?


Projects can establish their own policy on handling inactive members, as
long as they apply it consistently. It is not a problem to retain members
of the PMC who have become inactive, and it can make it easier for them to
stay in touch with the project if they choose to become active again.

Typically, PMC members who are no longer able to participate will resign
from the PMC. However, if a PMC chooses to remove one of its members
(without that member's request or consent), it must request the Board to
make that decision (which is typically done with a resolution at the
Board's next meeting). The PMC chair should send an email to the board@
mailing list detailing the request for removal and the justification the
PMC has for that removal, and copy the project's private@ list.

END QUOTE


= Summary
I believe that Apache Pulsar has the responsibility with respect to its
users to reflect the real number of people actively in the project - its
PMC members.

[VOTE] Apache Pulsar Adapters Release 2.11.0 Candidate 3

[Christophe Bornet]

This is the release candidate 3 for Apache Pulsar Adapters, version 2.11.0.

It fixes the following issues:
https://github.com/apache/pulsar-adapters/milestone/4?closed=1

*** Please download, test and vote on this release. This vote will
stay open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided
for convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-adapters-2.11.0-candidate-3/

SHA-512 checksums:
8272f786cb64caa34f8df0e07ea5ce7901e18c67ebf6f3c3cb6d887a7b0b44c6087279ddb5be8abbc62b56600a34060d556937ad9156266fd9d8c6314dc4dcd0
 apache-pulsar-adapters-2.11.0-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1208/

The tag to be voted upon:
v2.11.0-candidate-3 (3fbd9325920ff3eec3f32d539e1df81603e319f3)
https://github.com/apache/pulsar-adapters/releases/tag/v2.11.0-candidate-3

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build the
Pulsar Adapters code

Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic

[Yan Zhao]

Hi, Asaf. Tune the pip-186(https://github.com/apache/pulsar/issues/16569), 
please help to view it again. thanks!

On 2023/02/16 13:50:54 Yan Zhao wrote:
> > If understood correctly, every broker will have a consumer right? You will
> > use a fail-over subscription? The retry-topic is consumed by the same
> > subscription, same consumer?
> Yes, you are right, there is the case you mention. The deletion is 
> idempotent, I'm not sure if it's worth making it sync for it.
> 
> > In this very long mailing list thread, we have mentioned many fixes to be
> > done. Can you ping in the mailing list once you have managed to fix it all?
> 
> That's fine, I will push the new pip in two days. After the new pip pushing, 
> I will ping you.
> 

Re: [VOTE] Apache Pulsar Adapters Release 2.11.0 Candidate 2

[Enrico Olivelli]

+1 (binding)

- All tests pass on JDK17, builing from the source tarball
- checksums and signature looks great

Thanks for driving this release Christophe

Enrico

Il giorno gio 2 mar 2023 alle ore 12:29 Christophe Bornet
 ha scritto:
>
> This is the release candidate 2 for Apache Pulsar Adapters, version 2.11.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar-adapters/milestone/4?closed=1
>
> *** Please download, test and vote on this release. This vote will
> stay open for at least 72 hours ***
>
> Note that we are voting upon the source (tag), binaries are provided
> for convenience.
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-adapters-2.11.0-candidate-2/
>
> SHA-512 checksums:
> 0f46ca28ac0f1f53914969519ebc1c73bf00d8130a8d92f29333e3b22fc429ec7cdb3390b6b85d6c9aec9143a269f76d8f6a68706b802648babac4422d7c77ce
>  apache-pulsar-adapters-2.11.0-src.tar.gz
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachepulsar-1207/
>
> The tag to be voted upon:
> v2.11.0-candidate-2 (663663d791887b9884af8b6584591fa64f556af3)
> https://github.com/apache/pulsar-adapters/releases/tag/v2.11.0-candidate-2
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> Please download the source package, and follow the README to build the
> Pulsar Adapters code

[VOTE] Apache Pulsar Adapters Release 2.11.0 Candidate 2

[Christophe Bornet]

This is the release candidate 2 for Apache Pulsar Adapters, version 2.11.0.

It fixes the following issues:
https://github.com/apache/pulsar-adapters/milestone/4?closed=1

*** Please download, test and vote on this release. This vote will
stay open for at least 72 hours ***

Note that we are voting upon the source (tag), binaries are provided
for convenience.

Source and binary files:
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-adapters-2.11.0-candidate-2/

SHA-512 checksums:
0f46ca28ac0f1f53914969519ebc1c73bf00d8130a8d92f29333e3b22fc429ec7cdb3390b6b85d6c9aec9143a269f76d8f6a68706b802648babac4422d7c77ce
 apache-pulsar-adapters-2.11.0-src.tar.gz

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachepulsar-1207/

The tag to be voted upon:
v2.11.0-candidate-2 (663663d791887b9884af8b6584591fa64f556af3)
https://github.com/apache/pulsar-adapters/releases/tag/v2.11.0-candidate-2

Pulsar's KEYS file containing PGP keys we use to sign the release:
https://dist.apache.org/repos/dist/dev/pulsar/KEYS

Please download the source package, and follow the README to build the
Pulsar Adapters code

[VOTE]PIP-247: Notifications for partitions update.

[houxiaoyu]

Dear Community,

I would like to start a VOTE on "PIP-247: Notifications for partitions
update."

The proposal can be read at [0] and the discussion thread is available at
[1]

Voting will stay open for at least 48h.

[0] https://github.com/apache/pulsar/issues/19596
[1] https://lists.apache.org/thread/bcry0cz4z7kzot8pc4nhbktfv44xrk2y

Thanks,
Xiaoyu Hou

Re: [Discuss] PIP-248: Add backlog eviction metric

[太上玄元道君]

> I  think you should fix this explanation:

Thanks! I would like to copy the context you provide to the PIP motivation,
your description is more detailed, so developers don't have to go through
the code.

> Today the quota is checked periodically, right? So that's how the operator
> knows the cost in terms of I/O is limited.
> Now you are adding one additional I/O per collection, every 1 min by
> default. That's a lot perhaps. How long is the check interval today?

Actually, I don't want to introduce additional costs, I thought we
could cache its result, so that it won't introduce additional costs.
It may be that I did not make it clear in the PIP and caused this
misunderstanding, sorry.

> The user today can calculate quota used for size based limit, since there
> are two metrics that are exposed today on a topic level: "
> pulsar_storage_backlog_quota_limit" and "pulsar_storage_backlog_size". You
> can just divide the two to get a percentage.
> For the time-based limit, the only metric exposed today is quota itself ,
"
> pulsar_storage_backlog_quota_limit_time".

I only noticed `pulsar_storage_backlog_size` but missed
`pulsar_storage_backlog_quota_limit` and
`pulsar_storage_backlog_quota_limit_time`. Many thanks for your reminder.


So, in this condition, we already have the following topic-level metrics:
`pulsar_storage_backlog_size`: The total backlog size of the topics of this
topic owned by this broker (in bytes).
`pulsar_storage_backlog_quota_limit`: The total amount of the data in this
topic that limits the backlog quota (bytes).
`pulsar_storage_backlog_quota_limit_time`: The backlog quota limit in
time(seconds). (This metric does not exists in the doc, need to improve)


We just need to add a new metric named
`pulsar_storage_earliest_msg_publish_time_in_backlog` in the topic-level
that indicates the publish time of the earliest message in the backlog.
So users could get `pulsar_backlog_size_quota_used_percentage` by divide
`pulsar_storage_backlog_size ` and
`pulsar_storage_backlog_quota_limit`(`pulsar_storage_backlog_size` /
`pulsar_storage_backlog_quota_limit`),
and could get `pulsar_backlog_time_quota_used_percentage` by divide `now -
pulsar_storage_earliest_msg_publish_time_in_backlog` and
`pulsar_storage_backlog_quota_limit_time` (`now -
pulsar_storage_earliest_msg_publish_time_in_backlog` /
`pulsar_storage_backlog_quota_limit_time`).

The backlog quota time checker runs periodically, so we can cache its
result, so it won't lead to much costs.

Pulsar also exposed subscription-level  `backlogSize` and
`earliestMsgPublishTimeInBacklog` in Pulsar-Admin

if
`subscriptionBacklogSize` and `getEarliestTimeInBacklog` are true.
We can also expose `backlogQuotaLimiteSize` and `backlogQuotaLimitTime` of
the topic to PulsarAdmin.

After users receive the backlog alert from metrics alerting systems, they
can get the topic name, then, they can request Topics#getStats

to
get which subscriptions are in the huge backlog.

Thanks,
Tao Jiuming

Asaf Mesika  于2023年3月1日周三 23:42写道：

> >
> > Pulsar has 2 configurations for the backlog eviction
> > <
> https://pulsar.apache.org/docs/2.11.x/cookbooks-retention-expiry/#backlog-quotas
> >
> > : backlogQuotaDefaultLimitBytes and backlogQuotaDefaultLimitSecond.
> > By default, backlog eviction is disabled, and also, there is a field
> named
> > backlogQuotaMap in TopicPolicies
> > <
> https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/HierarchyTopicPolicies.java#L45
> >
> > /NamespaceSpacePolicies
> > <
> https://github.com/apache/pulsar/blob/master/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/Policies.java#L41>
> assists
> > in controlling Topic/Namespace level backlog quota.
> >
> > If topic backlog reaches the threshold of any item, backlog eviction will
> > be triggered, Pulsar will move subscription's cursor to skip
> unacknowledged
> > messages.
> >
> > Before backlog eviction happens, we don't have a metric to monitor how
> > long that it can reaches the threshold.
> >
>
> I  think you should fix this explanation:
>
> In Pulsar, a subscription maintains a state of message acknowledged. A
> subscription backlog is the set of messages which are unacknowledged.
> A subscription backlog size is the sum of size of unacknowledged messages
> (in bytes).
> A topic can have many subscriptions.
> A topic backlog is defined as the backlog size of the subscription which
> has the oldest unacknowledged message. Since acknowledged messages can be
> interleaved with unacknowledged messages, calculating the exact size of
> that subscription can be expensive as it requires I/O operations to read
> from the messages from the 

RE: Re: Introducer Pulsar admin api to pulsar-client-go

[Shen Eric]

Hi Zhangjian,

I am a PM from StreamNative and we also had some internal discussions related 
to this topic. Let me share our ongoing planning:

  *   We will extract the pulsar admin pkg from the pulsarctl to a separate 
open repo which will be called pulsar-admin-go under StreamNative.
  *   Will iterate the pulsar-admin-go library by adding more tests, 
documentation and may also update or fix the existing APIs.
  *   After the pulsar-admin-go library is stable, we will contribute this 
project to Apache Foundation.

On 2023/02/17 15:24:32 ZhangJian He wrote:
> Thank for StreamNative for willing to donate this project. This means we
> don't have to develop and maintain a set of HTTP code from scratch. My idea
> aligns with Yunze's, and separating it into a standalone pulsar-admin-go
> project would be better. The **pulsarctl** repo contains bookkeeper http
> call too. Maybe we can have a project bookkeeper-admin-go ?（it's a liitle
> going off-topic ）
>
> Thanks
> ZhangJian He
>
>
> On Fri, 17 Feb 2023 at 20:29, PengHui Li  wrote:
>
> > Hi Yunze,
> >
> > Yes, we can split it.
> > Both one repo with two modules or two repos works for me.
> >
> > The pulsarctl already have the admin API and CLI.
> > So I think we don’t need to develop another one.
> >
> > Best,
> > Penghui
> >
> > > On Feb 17, 2023, at 17:44, Yunze Xu 
> > wrote:
> > >
> > > Hi PengHui,
> > >
> > > Now I changed my mind a bit. Even if the pulsarctl was contributed to
> > > the Apache Foundation, I think we should also avoid adding it as the
> > > dependency. What we need is an API layer but not the CLI, while
> > > pulsarctl couples the API and CLI.
> > >
> > > At the moment, my expectation is:
> > > 1. Use a separate repo (e.g. pulsar-admin-go) to implement the admin
> > > APIs in Golang.
> > > 2. Depend this new repo in pulsarctl.
> > >
> > > Then we will have three Go projects:
> > > - pulsar-client-go: The Pulsar Go client APIs
> > > - pulsar-admin-go: The Pulsar Go admin APIs
> > > - pulsarctl: The admin CLI tool written in Go
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Fri, Feb 17, 2023 at 4:22 PM PengHui Li  wrote:
> > >>
> > >> I checked with Sijie today.
> > >> StreamNative can contribute the pulsarctl project to Apache Foundation.
> > >>
> > >> Regards,
> > >> Penghui
> > >>
> > >> On Fri, Feb 17, 2023 at 4:02 PM Enrico Olivelli 
> > wrote:
> > >>
> > >>> I agree to add an admin API to the go client, this would be very
> > helpful.
> > >>>
> > >>> Il giorno ven 17 feb 2023 alle ore 08:44 Zixuan Liu
> > >>>  ha scritto:
> > 
> >  Hi Zhangjian,
> > 
> >  This is a good idea to write the admin client by golang, but I don't
> >  suggest add the admin features to pulsar-go-client, it's better to
> > use a
> >  new repository to do that to separate dependencies.
> > 
> >  BTW, StreamNative has a pulsarctl [0] tool, which includes the admin
> > api.
> > 
> > >> It's better to reuse existing code rather than reinventing the
> > wheel.
> > 
> >  I aggred this point. If possible, we can integrate the pulsarctl to
> > this
> >  new project.
> > >>>
> > >>> We are talking about adding a client that calls a
> > >>> well defined and maintained REST API.
> > >>> It is better to have our implementation and not rely on third parties
> > >>> when it is possible.
> > >>> If there is a security issue in pulsarctl, how would we handle that ?
> > >>> Also the Pulsar community maintains the Pulsar API and this is the
> > >>> place where it is easier to keep the client up-to-date with the new
> > >>> APIs that we will develop,
> > >>> we can't wait for a third party project to implement our own APIs and
> > >>> wait for an upgrade (even if it is OSS, we cannot cut releases or have
> > >>> control over the release cycle)
> > >>>
> > >>>
> > >>> Enrico
> > >>>
> > >>>
> > >>>
> > 
> >  [0] - https://github.com/streamnative/pulsarctl
> > 
> >  Thanks,
> >  Zixuan
> > 
> > 
> >  ZhangJian He  于2023年2月17日周五 13:47写道：
> > 
> > > Separating dependencies is better. For example, I think
> > >>> Pulsar-admin-go can
> > > only have golang standard tls and http dependencies.
> > > But it seems impossible to have two go modules when publishing
> > packages
> > > using github.
> > >
> > >> Has anyone tried generating an admin client from our generated open
> > > api spec?
> > >
> > > I have attempted it, but it requires us to modify our Swagger file.
> > Our
> > > existing Swagger file can't generate HTTP clients directly. Perhaps
> > we
> > >>> can
> > > rewrite a unified and standardized Swagger file, and then generate
> > all
> > > code, including brokers, from there gradually.
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Fri, 17 Feb 2023 at 12:37, Yunze Xu  > >
> > > wrote:
> > >
> > >>> I notice that the Java Client and the Java Admin Client are
> > >>> separate
> > >>