Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar

2023-03-30 Thread Max Xu
Congratulations!  Qiang

Best,
Max Xu


On Wed, Mar 29, 2023 at 11:23 AM guo jiwei  wrote:

> Dear Community,
>
> We are thrilled to announce that Qiang Zhao
> (https://github.com/mattisonchao) has been invited and has accepted the
> role of member of the Apache Pulsar Project Management Committee (PMC).
>
> Qiang has been a vital asset to our community, consistently
> demonstrating his dedication and active participation through
> significant contributions. In addition to his technical contributions,
> Qiang also plays an important role in reviewing pull requests and
> ensuring the overall quality of our project. We look forward to his
> continued contributions.
>
> On behalf of the Pulsar PMC, we extend a warm welcome and
> congratulations to Qiang Zhao.
>
> Best regards
> Jiwei
>


Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar

2023-03-30 Thread Michael Marshall
Congratulations and welcome to the PMC!

- Michael

On Thu, Mar 30, 2023 at 8:21 PM PengHui Li  wrote:
>
> Congrats!
>
> Regards,
> Penghui
>
> On Thu, Mar 30, 2023 at 10:21 PM Zixuan Liu  wrote:
>
> > Congrats!
> >
> > Thanks,
> > Zixuan
> >
> > 太上玄元道君  于2023年3月30日周四 01:40写道:
> >
> > > Congrats!!
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > > > 2023年3月29日 23:51,Devin Bost  写道:
> > > >
> > > > Congrats!
> > > >
> > > > Devin G. Bost
> > > >
> > > >
> > > > On Wed, Mar 29, 2023 at 6:38 AM ZhangJian He 
> > wrote:
> > > >
> > > >> Congratulations!
> > > >>
> > > >> Thanks
> > > >> ZhangJian He
> > > >>
> > > >>
> > > >> On Wed, 29 Mar 2023 at 19:33, Haiting Jiang 
> > > >> wrote:
> > > >>
> > > >>> Congratulations!
> > > >>>
> > > >>>
> > > >>> Haiting
> > > >>>
> > > >>> On Wed, Mar 29, 2023 at 5:29 PM Cong Zhao 
> > wrote:
> > > 
> > >  Congrats! Qiang.
> > > 
> > > 
> > >  Thanks,
> > >  Cong Zhao
> > > 
> > >  On 2023/03/29 03:22:43 guo jiwei wrote:
> > > > Dear Community,
> > > >
> > > > We are thrilled to announce that Qiang Zhao
> > > > (https://github.com/mattisonchao) has been invited and has
> > accepted
> > > >>> the
> > > > role of member of the Apache Pulsar Project Management Committee
> > > >> (PMC).
> > > >
> > > > Qiang has been a vital asset to our community, consistently
> > > > demonstrating his dedication and active participation through
> > > > significant contributions. In addition to his technical
> > > >> contributions,
> > > > Qiang also plays an important role in reviewing pull requests and
> > > > ensuring the overall quality of our project. We look forward to his
> > > > continued contributions.
> > > >
> > > > On behalf of the Pulsar PMC, we extend a warm welcome and
> > > > congratulations to Qiang Zhao.
> > > >
> > > > Best regards
> > > > Jiwei
> > > >
> > > >>>
> > > >>
> > >
> > >
> >


Re: [VOTE] Pulsar Release 2.11.1 Candidate-1

2023-03-30 Thread PengHui Li
+1 (binding)

- Checked the signature
- Run standalone
- Run Pulsar perf
- Verified function and state function
- Verified Cassandra connector

Regards,
Penghui

On Mon, Mar 27, 2023 at 8:07 PM 丛搏  wrote:

> +1 (binding)
>
> system: mac os 12.6, Apple M1
> maven: 3.8.5
> java: OpenJDK 17.0.3
>
> - Checked the signature
> - Checked LICENSE
> - Start standalone with zookeeper stream storage
> - Publish and consume messages
> - Verified Function and State Function
> - Verified Cassandra connector
> - Build from the source package
> - Run a simple transaction check
>
> Thanks,
> Bo
>
> Yunze Xu  于2023年3月27日周一 12:01写道:
> >
> > +1 (binding)
> >
> > - Checked the signature
> > - Build from source (Java 17, Ubuntu 20.04 WSL2)
> > - Start standalone with both RocksDB and ZooKeeper
> > - Run basic end-to-end Pulsar tests and topic operations via pulsar-shell
> > - Run basic end-to-end Kafka tests with KoP 2.11.0.4
> >
> > Thanks,
> > Yunze
> >
> > On Wed, Mar 22, 2023 at 4:51 PM guo jiwei  wrote:
> > >
> > > This is the first release candidate for Apache Pulsar, version 2.11.1.
> > >
> > > This release contains 188 commits by 53 contributors.
> > > https://github.com/apache/pulsar/compare/v2.11.0...v2.11.1-candidate-1
> > >
> > > CI for this release candidate
> > > https://github.com/Technoboy-/pulsar/pull/28
> > >
> > > *** Please download, test and vote on this release. This vote will
> stay open
> > > for at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided
> for
> > > convenience.
> > >
> > > Source and binary files:
> > >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.1-candidate-1
> > >
> > > SHA-512 checksums:
> > >
> > >
> 7bd5c432fdb888dfcc2a1595efe29206545db535ea996aa4d1ff851e957cc88ce4b54fce912dde84baf8ed40217149a0acffad6c49f02f348721350e5ae895dd
> > >
> > >  ./apache-pulsar-2.11.1-bin.tar.gz
> > >
> > >
> > >
> 9e7bfac98e57a2a61216da77e48843bc4274828c9da1e695538d92a3ee929b52c4b0d2280feb73980d77f02c03f3c2dbc797673e69df19190ffee8e46760f305
> > >
> > >  ./apache-pulsar-2.11.1-src.tar.gz
> > >
> > > Maven staging repo:
> > >
> https://repository.apache.org/content/repositories/orgapachepulsar-1220/
> > >
> > > The tag to be voted upon:
> > > v2.11.1-candidate-1 (7cc41d7dec415acfeb1f96b68faaa2a80440e070)
> > > https://github.com/apache/pulsar/releases/tag/v2.11.1-candidate-1
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > >
> https://hub.docker.com/layers/mattison/pulsar-all/2.11.1-rc1/images/sha256-3d2c0bacbc6d34fa656466370744dedfd2b5a79df7a1178ea2fad27a6d6a
> > >
> > >
> https://hub.docker.com/layers/mattison/pulsar/2.11.1-rc1/images/sha256-1fda65d5637ad579ee36ab09448eb6ff3831583a9ef5b543bae78a565813cd95
> > >
> > > Please download the source package, and follow the
> > > release-candidate-validation doc to build
> > > and run the Pulsar standalone service.
> > > https://pulsar.apache.org/contribute/validate-release-candidate
> > >
> > > Since the metadata store is changed from ZK to RocksDB, the
> > > verification of the `stateful functions` needs to set the parameter
> > > "export PULSAR_STANDALONE_USE_ZOOKEEPER=1"
> > >
> > >
> > > Regards
> > > Jiwei Guo (Tboy)
>


Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread Nitin Goyal
+1 (non-binding)

On Fri, 31 Mar, 2023, 08:41 Jun Ma,  wrote:

> +1.
>
> Besides using a single source to lift the review efficiency, adding
> control over the design documents is also a good practice from the project
> management perspective.
>
>
> Best,
> Jun
> 
> From: Yunze Xu 
> Sent: Friday, March 31, 2023 10:44
> To: dev@pulsar.apache.org 
> Subject: Re: [DISCUSS] We must change the way we review PIPs
>
> +1 to me. Once the discussion thread of a PIP became too long, it
> would be hard to continue the discussion.
>
> Thanks,
> Yunze
>
> On Fri, Mar 31, 2023 at 9:13 AM PengHui Li  wrote:
> >
> > +1 for creating a folder named "pip" in the main pulsar repo
> > So far, it is good enough to solve the problems we've had.
> >
> > If it is really worth moving to another repo in the future.
> > We can move it maybe 3, 5 years later.
> >
> > Thanks,
> > Penghui
> >
> >
> > On Fri, Mar 31, 2023 at 8:29 AM tison  wrote:
> >
> > > Hi Asaf,
> > >
> > > Thanks for starting this thread!
> > >
> > > I have similar thoughts on using a single source for reviewing PIPs.
> GH PRs
> > > are good for conversation, although multiple conversations are still
> hard
> > > to follow (which can be natural)
> > >
> > > Here is how Rust does it[1] - a self-documented RFC repo + review PRs +
> > > tracking issue on the main repo once accepted.
> > >
> > > Here is what I designed for TiDB[2][3]; proposals are placed in a
> folder
> > > under the main repo.
> > >
> > > We don't have to follow other community's ways, but these two practices
> > > seem good to read.
> > >
> > > And, to follow the ASF voting strategy[4][5], we may still need a
> standard
> > > vote mail thread and document the result on the mailing list.
> > >
> > > Best,
> > > tison.
> > >
> > > [1] https://github.com/rust-lang/rfcs
> > > [2] https://github.com/pingcap/tidb/tree/master/docs/design
> > > [3]
> > >
> > >
> https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/make-a-proposal.html
> > > [4] https://www.apache.org/foundation/voting.html
> > > [5] https://www.apache.org/dev/pmc.html#mailing-list-naming-policy
> > >
> > >
> > > Girish Sharma  于2023年3月31日周五 04:33写道:
> > >
> > > > +1 (non-binding .. ? )
> > > > I've already commented a couple of times (here and there) that the
> > > process
> > > > needs to be consolidated at a single place. This is a good and
> detailed
> > > > approach.
> > > > Not sure if there is a historical context behind keeping the
> discussion
> > > in
> > > > dev mailing list..
> > > >
> > > > Regards
> > > >
> > > > On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > In the last 2 months, I've increased my PIP review time (I do it in
> > > > > cycles), and reviewed quite a few PIPs.
> > > > >
> > > > > My conclusion as a result of that:
> > > > >
> > > > > It's nearly impossible to review PIPs using a mailing list.
> > > > > We must fix it soon.
> > > > >
> > > > > *Why?*
> > > > > 1. Let's say you review the PIP and find 10 issues. Once you quote
> and
> > > > > comment on those ten points, you basically started 10 threads of
> > > > > conversations.
> > > > > After 2-3 ping pongs with quotes of quotes of quotes, it takes you
> > > > forever
> > > > > to read each thread properly. You find your CTRL-F to search to
> find
> > > your
> > > > > original quote, and reply. Load it up again in your head,
> switching to
> > > > the
> > > > > PIP tab to read it again.
> > > > > After 10 ping pongs, it becomes almost an impossible mission.
> > > > >
> > > > > I can say I'm 75% tired just from the process, not from the review
> > > > itself.
> > > > >
> > > > > 2. It's non collaborative by design.
> > > > > After 10 ping pongs, the ability of someone to come and join the
> > > > discussion
> > > > > is 0. They need to go through so many replies, which are half
> quotes,
> > > > find
> > > > > the original reply, and browse to the PIP.
> > > > > It's no wonder people drop off the PIP review once we cross 5-6
> > > replies.
> > > > > It's no wonder, nobody joins after 10 replies.
> > > > >
> > > > > 3. It's not open to the public. Non collaborative.
> > > > > You can't just get a link, and join the review. Not only because
> of (1)
> > > > and
> > > > > (2). You need to join the mailing list. You don't get the past
> emails
> > > to
> > > > > reply. Just joining the list is a high enough bar for many people.
> > > > > I personally participated in review of proposals in OpenTelemetry
> in
> > > the
> > > > > last 6 months, even though I'm just an occasional user.  Why? They
> were
> > > > > conducted on GitHub PR , so it was easy for me - click a link and
> > > reply.
> > > > >
> > > > > 4. All over the place
> > > > > Sometimes people comment on the GitHub issue.
> > > > > Sometimes on the mailing list.
> > > > > Not a single place.
> > > > >
> > > > > 5. No history.
> > > > > Ok, finally the author was convinced. I can't see just the changes.
> > > 

Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread Jun Ma
+1.

Besides using a single source to lift the review efficiency, adding control 
over the design documents is also a good practice from the project management 
perspective.


Best,
Jun

From: Yunze Xu 
Sent: Friday, March 31, 2023 10:44
To: dev@pulsar.apache.org 
Subject: Re: [DISCUSS] We must change the way we review PIPs

+1 to me. Once the discussion thread of a PIP became too long, it
would be hard to continue the discussion.

Thanks,
Yunze

On Fri, Mar 31, 2023 at 9:13 AM PengHui Li  wrote:
>
> +1 for creating a folder named "pip" in the main pulsar repo
> So far, it is good enough to solve the problems we've had.
>
> If it is really worth moving to another repo in the future.
> We can move it maybe 3, 5 years later.
>
> Thanks,
> Penghui
>
>
> On Fri, Mar 31, 2023 at 8:29 AM tison  wrote:
>
> > Hi Asaf,
> >
> > Thanks for starting this thread!
> >
> > I have similar thoughts on using a single source for reviewing PIPs. GH PRs
> > are good for conversation, although multiple conversations are still hard
> > to follow (which can be natural)
> >
> > Here is how Rust does it[1] - a self-documented RFC repo + review PRs +
> > tracking issue on the main repo once accepted.
> >
> > Here is what I designed for TiDB[2][3]; proposals are placed in a folder
> > under the main repo.
> >
> > We don't have to follow other community's ways, but these two practices
> > seem good to read.
> >
> > And, to follow the ASF voting strategy[4][5], we may still need a standard
> > vote mail thread and document the result on the mailing list.
> >
> > Best,
> > tison.
> >
> > [1] https://github.com/rust-lang/rfcs
> > [2] https://github.com/pingcap/tidb/tree/master/docs/design
> > [3]
> >
> > https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/make-a-proposal.html
> > [4] https://www.apache.org/foundation/voting.html
> > [5] https://www.apache.org/dev/pmc.html#mailing-list-naming-policy
> >
> >
> > Girish Sharma  于2023年3月31日周五 04:33写道:
> >
> > > +1 (non-binding .. ? )
> > > I've already commented a couple of times (here and there) that the
> > process
> > > needs to be consolidated at a single place. This is a good and detailed
> > > approach.
> > > Not sure if there is a historical context behind keeping the discussion
> > in
> > > dev mailing list..
> > >
> > > Regards
> > >
> > > On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > In the last 2 months, I've increased my PIP review time (I do it in
> > > > cycles), and reviewed quite a few PIPs.
> > > >
> > > > My conclusion as a result of that:
> > > >
> > > > It's nearly impossible to review PIPs using a mailing list.
> > > > We must fix it soon.
> > > >
> > > > *Why?*
> > > > 1. Let's say you review the PIP and find 10 issues. Once you quote and
> > > > comment on those ten points, you basically started 10 threads of
> > > > conversations.
> > > > After 2-3 ping pongs with quotes of quotes of quotes, it takes you
> > > forever
> > > > to read each thread properly. You find your CTRL-F to search to find
> > your
> > > > original quote, and reply. Load it up again in your head, switching to
> > > the
> > > > PIP tab to read it again.
> > > > After 10 ping pongs, it becomes almost an impossible mission.
> > > >
> > > > I can say I'm 75% tired just from the process, not from the review
> > > itself.
> > > >
> > > > 2. It's non collaborative by design.
> > > > After 10 ping pongs, the ability of someone to come and join the
> > > discussion
> > > > is 0. They need to go through so many replies, which are half quotes,
> > > find
> > > > the original reply, and browse to the PIP.
> > > > It's no wonder people drop off the PIP review once we cross 5-6
> > replies.
> > > > It's no wonder, nobody joins after 10 replies.
> > > >
> > > > 3. It's not open to the public. Non collaborative.
> > > > You can't just get a link, and join the review. Not only because of (1)
> > > and
> > > > (2). You need to join the mailing list. You don't get the past emails
> > to
> > > > reply. Just joining the list is a high enough bar for many people.
> > > > I personally participated in review of proposals in OpenTelemetry in
> > the
> > > > last 6 months, even though I'm just an occasional user.  Why? They were
> > > > conducted on GitHub PR , so it was easy for me - click a link and
> > reply.
> > > >
> > > > 4. All over the place
> > > > Sometimes people comment on the GitHub issue.
> > > > Sometimes on the mailing list.
> > > > Not a single place.
> > > >
> > > > 5. No history.
> > > > Ok, finally the author was convinced. I can't see just the changes.
> > They
> > > > need to explicitly tell me something was changed.
> > > >
> > > > 6. Delete All.
> > > > They can go crazy, after 1 year, edit the GitHub issue , delete all the
> > > > text and write "Kafka is the king". No history, nobody can stop them.
> > > It's
> > > > their issue.
> > > >
> > > > 7. Show me all the approved PIPs
> > > > Hard to track it today, 

Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread houxiaoyu
Big +1(non-binding) to me

Thanks,
Hou Xiaoyu

Yunze Xu  于2023年3月31日周五 10:45写道:

> +1 to me. Once the discussion thread of a PIP became too long, it
> would be hard to continue the discussion.
>
> Thanks,
> Yunze
>
> On Fri, Mar 31, 2023 at 9:13 AM PengHui Li  wrote:
> >
> > +1 for creating a folder named "pip" in the main pulsar repo
> > So far, it is good enough to solve the problems we've had.
> >
> > If it is really worth moving to another repo in the future.
> > We can move it maybe 3, 5 years later.
> >
> > Thanks,
> > Penghui
> >
> >
> > On Fri, Mar 31, 2023 at 8:29 AM tison  wrote:
> >
> > > Hi Asaf,
> > >
> > > Thanks for starting this thread!
> > >
> > > I have similar thoughts on using a single source for reviewing PIPs.
> GH PRs
> > > are good for conversation, although multiple conversations are still
> hard
> > > to follow (which can be natural)
> > >
> > > Here is how Rust does it[1] - a self-documented RFC repo + review PRs +
> > > tracking issue on the main repo once accepted.
> > >
> > > Here is what I designed for TiDB[2][3]; proposals are placed in a
> folder
> > > under the main repo.
> > >
> > > We don't have to follow other community's ways, but these two practices
> > > seem good to read.
> > >
> > > And, to follow the ASF voting strategy[4][5], we may still need a
> standard
> > > vote mail thread and document the result on the mailing list.
> > >
> > > Best,
> > > tison.
> > >
> > > [1] https://github.com/rust-lang/rfcs
> > > [2] https://github.com/pingcap/tidb/tree/master/docs/design
> > > [3]
> > >
> > >
> https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/make-a-proposal.html
> > > [4] https://www.apache.org/foundation/voting.html
> > > [5] https://www.apache.org/dev/pmc.html#mailing-list-naming-policy
> > >
> > >
> > > Girish Sharma  于2023年3月31日周五 04:33写道:
> > >
> > > > +1 (non-binding .. ? )
> > > > I've already commented a couple of times (here and there) that the
> > > process
> > > > needs to be consolidated at a single place. This is a good and
> detailed
> > > > approach.
> > > > Not sure if there is a historical context behind keeping the
> discussion
> > > in
> > > > dev mailing list..
> > > >
> > > > Regards
> > > >
> > > > On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > In the last 2 months, I've increased my PIP review time (I do it in
> > > > > cycles), and reviewed quite a few PIPs.
> > > > >
> > > > > My conclusion as a result of that:
> > > > >
> > > > > It's nearly impossible to review PIPs using a mailing list.
> > > > > We must fix it soon.
> > > > >
> > > > > *Why?*
> > > > > 1. Let's say you review the PIP and find 10 issues. Once you quote
> and
> > > > > comment on those ten points, you basically started 10 threads of
> > > > > conversations.
> > > > > After 2-3 ping pongs with quotes of quotes of quotes, it takes you
> > > > forever
> > > > > to read each thread properly. You find your CTRL-F to search to
> find
> > > your
> > > > > original quote, and reply. Load it up again in your head,
> switching to
> > > > the
> > > > > PIP tab to read it again.
> > > > > After 10 ping pongs, it becomes almost an impossible mission.
> > > > >
> > > > > I can say I'm 75% tired just from the process, not from the review
> > > > itself.
> > > > >
> > > > > 2. It's non collaborative by design.
> > > > > After 10 ping pongs, the ability of someone to come and join the
> > > > discussion
> > > > > is 0. They need to go through so many replies, which are half
> quotes,
> > > > find
> > > > > the original reply, and browse to the PIP.
> > > > > It's no wonder people drop off the PIP review once we cross 5-6
> > > replies.
> > > > > It's no wonder, nobody joins after 10 replies.
> > > > >
> > > > > 3. It's not open to the public. Non collaborative.
> > > > > You can't just get a link, and join the review. Not only because
> of (1)
> > > > and
> > > > > (2). You need to join the mailing list. You don't get the past
> emails
> > > to
> > > > > reply. Just joining the list is a high enough bar for many people.
> > > > > I personally participated in review of proposals in OpenTelemetry
> in
> > > the
> > > > > last 6 months, even though I'm just an occasional user.  Why? They
> were
> > > > > conducted on GitHub PR , so it was easy for me - click a link and
> > > reply.
> > > > >
> > > > > 4. All over the place
> > > > > Sometimes people comment on the GitHub issue.
> > > > > Sometimes on the mailing list.
> > > > > Not a single place.
> > > > >
> > > > > 5. No history.
> > > > > Ok, finally the author was convinced. I can't see just the changes.
> > > They
> > > > > need to explicitly tell me something was changed.
> > > > >
> > > > > 6. Delete All.
> > > > > They can go crazy, after 1 year, edit the GitHub issue , delete
> all the
> > > > > text and write "Kafka is the king". No history, nobody can stop
> them.
> > > > It's
> > > > > their issue.
> > > > >
> > > > > 7. Show me all the approved PIPs
> > > > > 

Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar

2023-03-30 Thread Jun Ma
Congrats, Qiang!

Cheers,
Jun


Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread Yunze Xu
+1 to me. Once the discussion thread of a PIP became too long, it
would be hard to continue the discussion.

Thanks,
Yunze

On Fri, Mar 31, 2023 at 9:13 AM PengHui Li  wrote:
>
> +1 for creating a folder named "pip" in the main pulsar repo
> So far, it is good enough to solve the problems we've had.
>
> If it is really worth moving to another repo in the future.
> We can move it maybe 3, 5 years later.
>
> Thanks,
> Penghui
>
>
> On Fri, Mar 31, 2023 at 8:29 AM tison  wrote:
>
> > Hi Asaf,
> >
> > Thanks for starting this thread!
> >
> > I have similar thoughts on using a single source for reviewing PIPs. GH PRs
> > are good for conversation, although multiple conversations are still hard
> > to follow (which can be natural)
> >
> > Here is how Rust does it[1] - a self-documented RFC repo + review PRs +
> > tracking issue on the main repo once accepted.
> >
> > Here is what I designed for TiDB[2][3]; proposals are placed in a folder
> > under the main repo.
> >
> > We don't have to follow other community's ways, but these two practices
> > seem good to read.
> >
> > And, to follow the ASF voting strategy[4][5], we may still need a standard
> > vote mail thread and document the result on the mailing list.
> >
> > Best,
> > tison.
> >
> > [1] https://github.com/rust-lang/rfcs
> > [2] https://github.com/pingcap/tidb/tree/master/docs/design
> > [3]
> >
> > https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/make-a-proposal.html
> > [4] https://www.apache.org/foundation/voting.html
> > [5] https://www.apache.org/dev/pmc.html#mailing-list-naming-policy
> >
> >
> > Girish Sharma  于2023年3月31日周五 04:33写道:
> >
> > > +1 (non-binding .. ? )
> > > I've already commented a couple of times (here and there) that the
> > process
> > > needs to be consolidated at a single place. This is a good and detailed
> > > approach.
> > > Not sure if there is a historical context behind keeping the discussion
> > in
> > > dev mailing list..
> > >
> > > Regards
> > >
> > > On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > In the last 2 months, I've increased my PIP review time (I do it in
> > > > cycles), and reviewed quite a few PIPs.
> > > >
> > > > My conclusion as a result of that:
> > > >
> > > > It's nearly impossible to review PIPs using a mailing list.
> > > > We must fix it soon.
> > > >
> > > > *Why?*
> > > > 1. Let's say you review the PIP and find 10 issues. Once you quote and
> > > > comment on those ten points, you basically started 10 threads of
> > > > conversations.
> > > > After 2-3 ping pongs with quotes of quotes of quotes, it takes you
> > > forever
> > > > to read each thread properly. You find your CTRL-F to search to find
> > your
> > > > original quote, and reply. Load it up again in your head, switching to
> > > the
> > > > PIP tab to read it again.
> > > > After 10 ping pongs, it becomes almost an impossible mission.
> > > >
> > > > I can say I'm 75% tired just from the process, not from the review
> > > itself.
> > > >
> > > > 2. It's non collaborative by design.
> > > > After 10 ping pongs, the ability of someone to come and join the
> > > discussion
> > > > is 0. They need to go through so many replies, which are half quotes,
> > > find
> > > > the original reply, and browse to the PIP.
> > > > It's no wonder people drop off the PIP review once we cross 5-6
> > replies.
> > > > It's no wonder, nobody joins after 10 replies.
> > > >
> > > > 3. It's not open to the public. Non collaborative.
> > > > You can't just get a link, and join the review. Not only because of (1)
> > > and
> > > > (2). You need to join the mailing list. You don't get the past emails
> > to
> > > > reply. Just joining the list is a high enough bar for many people.
> > > > I personally participated in review of proposals in OpenTelemetry in
> > the
> > > > last 6 months, even though I'm just an occasional user.  Why? They were
> > > > conducted on GitHub PR , so it was easy for me - click a link and
> > reply.
> > > >
> > > > 4. All over the place
> > > > Sometimes people comment on the GitHub issue.
> > > > Sometimes on the mailing list.
> > > > Not a single place.
> > > >
> > > > 5. No history.
> > > > Ok, finally the author was convinced. I can't see just the changes.
> > They
> > > > need to explicitly tell me something was changed.
> > > >
> > > > 6. Delete All.
> > > > They can go crazy, after 1 year, edit the GitHub issue , delete all the
> > > > text and write "Kafka is the king". No history, nobody can stop them.
> > > It's
> > > > their issue.
> > > >
> > > > 7. Show me all the approved PIPs
> > > > Hard to track it today, hard to maintain it updated.
> > > >
> > > > 8. Resolved comments
> > > > Even though you managed to read all 35 replies so far, in reply 36 you
> > > see
> > > > the author agreed to all 8 out of 10 suggestions. You have no idea of
> > > > knowing that in advance. You just wasted 1 hour.
> > > >
> > > >
> > > > *What do I suggest?*
> > > >
> > > > PR is 

Re: [Python] Should we make the schema default compatible with Java client?

2023-03-30 Thread 丛搏
Hi, Yunze:

+1

> Just checked this thread and found I didn't paste this issue:
> https://github.com/apache/pulsar-client-python/issues/108. You can see
> the schema compatibility strategy is FORWARD, then the sorted schema
> from the Java client overwrote the unsorted schema from the Python
> client. However, the Python consumer that uses the old schema failed
> to decode the message of the new schema.

if this changes the default behavior, the user upgrading the python
client will register one more schema. This is a breaking change, so the
old users need to change these using python schema code when they
upgrade the python client. We need to note it in the release note.

Thanks
Bo
>
> My goal is to make the Python client act the same as the Java client
> since the next formal release. Regarding how the broker processes it,
> I think it's another thing to be fixed.
>
> Thanks,
> Yunze
>
> On Thu, Mar 30, 2023 at 8:42 PM 丛搏  wrote:
> >
> > Hi, Yunze:
> >
> > > Regarding the 1st question, yes, that's why I open this thread to
> > > discuss. If we change these default values, the behavior of new Python
> > > clients will be like the Java client. In addition, it actually reverts
> > > the breaking change brought in #12232.
> >
> > I also kind of forget why we have #12232 to change the default behavior
> > Maybe the python2 and python3 order rule is different.
> >
> > If we change the order is the default value, for every topic that uses
> > python client will register a new schema. Will it register a new
> > schema? Maybe we should add a special logic in the broker to
> > check the python client version and make it will not register
> > a new schema. Otherwise, the impact may still be quite large.
> >
> > Thanks,
> > Bo
> > >
> > > Regarding the 2nd question, yes, they are both sorted in alphabetical
> > > order. I don't know the behavior of the .NET clients, for C++, Golang,
> > > Node.js clients, they all do not support generating schema definition
> > > from a DTO.
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Thu, Mar 30, 2023 at 10:14 AM 丛搏  wrote:
> > > >
> > > > Hi, Yunze :
> > > >
> > > > 1. If the changes may cause some compatibility issues.
> > > > How do we solve the compatibility issues? It may be a
> > > > breaking change.
> > > >
> > > > 2. Another question is if sorting is enabled by default,
> > > > is the sorting rule the same as java or other clients?
> > > >
> > > > Putting aside the above two problems, I think it is
> > > > good to be consistent with other clients.
> > > >
> > > > Thanks,
> > > > Bo
> > > >
> > > > Eric Hare  于2023年3月29日周三 22:42写道:
> > > > >
> > > > > +1 - i think keeping the `_sorted_fields` and `_required` defaults 
> > > > > consistent between the clients is the way to go.
> > > > >
> > > > > > On Mar 29, 2023, at 7:09 AM, Yunze Xu 
> > > > > >  wrote:
> > > > > >
> > > > > > I found the Python client has two options to control the behavior:
> > > > > > 1. Set `_sorted_fields`. It's false by default in the Python client,
> > > > > > but it's true in the Java client. i.e. the Java client sorts all
> > > > > > fields by default.
> > > > > > 2. Set `_required`. It's false by default for all types in the 
> > > > > > Python
> > > > > > client, but it's only false for the string type in the Java client.
> > > > > >
> > > > > > i.e. given the following Java class:
> > > > > >
> > > > > > ```java
> > > > > > class User {
> > > > > >String name;
> > > > > >int age;
> > > > > >double score;
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > We have to give the following definition in Python:
> > > > > >
> > > > > > ```python
> > > > > > class User(Record):
> > > > > >_sorted_fields = True
> > > > > >name = String()
> > > > > >age = Integer(required=True)
> > > > > >score = Double(required=True)
> > > > > > ```
> > > > > >
> > > > > > I see https://github.com/apache/pulsar/pull/12232 adds the
> > > > > > `_sorted_fields` field and disables the field sort by default. It
> > > > > > breaks compatibility with the Java client.
> > > > > >
> > > > > > IMO, we should make `_sorted_fields` true by default and `_required`
> > > > > > true for all types other than `String` by default.
> > > > > >
> > > > > > Thanks,
> > > > > > Yunze
> > > > > >
> > > > > > On Wed, Mar 29, 2023 at 4:00 PM Yunze Xu  
> > > > > > wrote:
> > > > > >>
> > > > > >> Hi all,
> > > > > >>
> > > > > >> Recently I found the default generated schema definition in the 
> > > > > >> Python
> > > > > >> client is different from the Java client, which leads to some
> > > > > >> unexpected behavior.
> > > > > >>
> > > > > >> For example, given the following class definition in Python:
> > > > > >>
> > > > > >> ```python
> > > > > >> class Data(Record):
> > > > > >>i = Integer()
> > > > > >> ```
> > > > > >>
> > > > > >> The type of `i` field is a union: "type": ["null", "int"]
> > > > > >>
> > > > > >> While given the following class definition in Java:
> > > > > >>
> > > > > >> 

Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar

2023-03-30 Thread PengHui Li
Congrats!

Regards,
Penghui

On Thu, Mar 30, 2023 at 10:21 PM Zixuan Liu  wrote:

> Congrats!
>
> Thanks,
> Zixuan
>
> 太上玄元道君  于2023年3月30日周四 01:40写道:
>
> > Congrats!!
> >
> > Thanks,
> > Tao Jiuming
> >
> > > 2023年3月29日 23:51,Devin Bost  写道:
> > >
> > > Congrats!
> > >
> > > Devin G. Bost
> > >
> > >
> > > On Wed, Mar 29, 2023 at 6:38 AM ZhangJian He 
> wrote:
> > >
> > >> Congratulations!
> > >>
> > >> Thanks
> > >> ZhangJian He
> > >>
> > >>
> > >> On Wed, 29 Mar 2023 at 19:33, Haiting Jiang 
> > >> wrote:
> > >>
> > >>> Congratulations!
> > >>>
> > >>>
> > >>> Haiting
> > >>>
> > >>> On Wed, Mar 29, 2023 at 5:29 PM Cong Zhao 
> wrote:
> > 
> >  Congrats! Qiang.
> > 
> > 
> >  Thanks,
> >  Cong Zhao
> > 
> >  On 2023/03/29 03:22:43 guo jiwei wrote:
> > > Dear Community,
> > >
> > > We are thrilled to announce that Qiang Zhao
> > > (https://github.com/mattisonchao) has been invited and has
> accepted
> > >>> the
> > > role of member of the Apache Pulsar Project Management Committee
> > >> (PMC).
> > >
> > > Qiang has been a vital asset to our community, consistently
> > > demonstrating his dedication and active participation through
> > > significant contributions. In addition to his technical
> > >> contributions,
> > > Qiang also plays an important role in reviewing pull requests and
> > > ensuring the overall quality of our project. We look forward to his
> > > continued contributions.
> > >
> > > On behalf of the Pulsar PMC, we extend a warm welcome and
> > > congratulations to Qiang Zhao.
> > >
> > > Best regards
> > > Jiwei
> > >
> > >>>
> > >>
> >
> >
>


Re: [DISCUSS] Add checklist for PMC binding vote of PIP

2023-03-30 Thread PengHui Li
It looks like we can try to add a new section to
https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md
like "Review the proposal" and it is not only for PMCs, all the reviewers
can follow the checklist
to cast a solemn vote.

And I totally support the motivation of this discussion.

Regards,
Penghui

On Fri, Mar 31, 2023 at 4:46 AM Asaf Mesika  wrote:

> Hi,
>
> When you read last year's PIPs, many lack background information, hard to
> read and understand even if you know pulsar in and out.
>
> First step to fix was to change the PIP is structured:
> https://github.com/apache/pulsar/pull/19832
>
> In my opinion, when someone votes "+1" and it's binding, they basically
> take the responsibility to say:
>
> * I read the PIP fully.
> * A person having basic Pulsar user knowledge, can read the PIP and fully
> understand it
>   Why? Since it contains all background information necessary to
> understand the problem and the solution
>It is written in a coherent and easy to understand way.
> * I validated the solution technically and can vouch for it.
>Examples:
>The PIP adds schema compatibility rules for Protobuf Native.
>  I learned / know protobuf well.
>  I validated the rules written containing all rules needed and
> not containing wrong rules, or missing rules.
>
>The PIP adds new OpenID Connect authentication.
>   I learned / know Authentication in Pulsar.
>I learned / know OpenID connect
>I validated the solution is architecturally correct and
> sound.
>
> Basically the PMC member voting +1 on it, basically acts as Tech Lead of
> Pulsar for this PIP.
> It's a very big responsibility.
> It's the only way to ensure Pulsar architecture won't go haywire over the
> next few years.
>
> Yes, it will slow the process down.
> Yes, it will be harder to find people to review it like that.
>
> But, it will raise the bar for PIPs and for Pulsar architecture overall.
> IMO we need that, and it's customary.
>
> *My suggestion*
> When PMC member replies to vote, it will look like this:
>
> "
> +1 (binding)
>
> [v] PIP has all sections detailed in the PIP template (Background,
> motivation, etc.)
> [v] A person having basic Pulsar user knowledge, can read the PIP and fully
> understand it
> [v] I read PIP and validated it technically
> "
>
> or
> "
> -1 (binding)
>
> I think this PIP needs:
> ...
> "
>
> Thanks,
>
> Asaf
>


Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread PengHui Li
+1 for creating a folder named "pip" in the main pulsar repo
So far, it is good enough to solve the problems we've had.

If it is really worth moving to another repo in the future.
We can move it maybe 3, 5 years later.

Thanks,
Penghui


On Fri, Mar 31, 2023 at 8:29 AM tison  wrote:

> Hi Asaf,
>
> Thanks for starting this thread!
>
> I have similar thoughts on using a single source for reviewing PIPs. GH PRs
> are good for conversation, although multiple conversations are still hard
> to follow (which can be natural)
>
> Here is how Rust does it[1] - a self-documented RFC repo + review PRs +
> tracking issue on the main repo once accepted.
>
> Here is what I designed for TiDB[2][3]; proposals are placed in a folder
> under the main repo.
>
> We don't have to follow other community's ways, but these two practices
> seem good to read.
>
> And, to follow the ASF voting strategy[4][5], we may still need a standard
> vote mail thread and document the result on the mailing list.
>
> Best,
> tison.
>
> [1] https://github.com/rust-lang/rfcs
> [2] https://github.com/pingcap/tidb/tree/master/docs/design
> [3]
>
> https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/make-a-proposal.html
> [4] https://www.apache.org/foundation/voting.html
> [5] https://www.apache.org/dev/pmc.html#mailing-list-naming-policy
>
>
> Girish Sharma  于2023年3月31日周五 04:33写道:
>
> > +1 (non-binding .. ? )
> > I've already commented a couple of times (here and there) that the
> process
> > needs to be consolidated at a single place. This is a good and detailed
> > approach.
> > Not sure if there is a historical context behind keeping the discussion
> in
> > dev mailing list..
> >
> > Regards
> >
> > On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika 
> wrote:
> >
> > > Hi all,
> > >
> > > In the last 2 months, I've increased my PIP review time (I do it in
> > > cycles), and reviewed quite a few PIPs.
> > >
> > > My conclusion as a result of that:
> > >
> > > It's nearly impossible to review PIPs using a mailing list.
> > > We must fix it soon.
> > >
> > > *Why?*
> > > 1. Let's say you review the PIP and find 10 issues. Once you quote and
> > > comment on those ten points, you basically started 10 threads of
> > > conversations.
> > > After 2-3 ping pongs with quotes of quotes of quotes, it takes you
> > forever
> > > to read each thread properly. You find your CTRL-F to search to find
> your
> > > original quote, and reply. Load it up again in your head, switching to
> > the
> > > PIP tab to read it again.
> > > After 10 ping pongs, it becomes almost an impossible mission.
> > >
> > > I can say I'm 75% tired just from the process, not from the review
> > itself.
> > >
> > > 2. It's non collaborative by design.
> > > After 10 ping pongs, the ability of someone to come and join the
> > discussion
> > > is 0. They need to go through so many replies, which are half quotes,
> > find
> > > the original reply, and browse to the PIP.
> > > It's no wonder people drop off the PIP review once we cross 5-6
> replies.
> > > It's no wonder, nobody joins after 10 replies.
> > >
> > > 3. It's not open to the public. Non collaborative.
> > > You can't just get a link, and join the review. Not only because of (1)
> > and
> > > (2). You need to join the mailing list. You don't get the past emails
> to
> > > reply. Just joining the list is a high enough bar for many people.
> > > I personally participated in review of proposals in OpenTelemetry in
> the
> > > last 6 months, even though I'm just an occasional user.  Why? They were
> > > conducted on GitHub PR , so it was easy for me - click a link and
> reply.
> > >
> > > 4. All over the place
> > > Sometimes people comment on the GitHub issue.
> > > Sometimes on the mailing list.
> > > Not a single place.
> > >
> > > 5. No history.
> > > Ok, finally the author was convinced. I can't see just the changes.
> They
> > > need to explicitly tell me something was changed.
> > >
> > > 6. Delete All.
> > > They can go crazy, after 1 year, edit the GitHub issue , delete all the
> > > text and write "Kafka is the king". No history, nobody can stop them.
> > It's
> > > their issue.
> > >
> > > 7. Show me all the approved PIPs
> > > Hard to track it today, hard to maintain it updated.
> > >
> > > 8. Resolved comments
> > > Even though you managed to read all 35 replies so far, in reply 36 you
> > see
> > > the author agreed to all 8 out of 10 suggestions. You have no idea of
> > > knowing that in advance. You just wasted 1 hour.
> > >
> > >
> > > *What do I suggest?*
> > >
> > > PR is the main tool we have that allows multiple threaded discussion.
> > > Git provides history. You can't delete it without approval from PMC
> > > members.
> > >
> > > 1. We'll create a folder named "pip" in the pulsar main repo. It will
> > > contain one markdown file for each PIP. The file will be named
> > > "pip-xxx.md".I will write below how to obtain XXX before you start.
> > > 2. To create a PIP, you grab "pip/template.md" and use it to 

Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread Christophe Bornet
Big +1 for me

Le jeu. 30 mars 2023 à 22:27, Asaf Mesika  a écrit :
>
> Hi all,
>
> In the last 2 months, I've increased my PIP review time (I do it in
> cycles), and reviewed quite a few PIPs.
>
> My conclusion as a result of that:
>
> It's nearly impossible to review PIPs using a mailing list.
> We must fix it soon.
>
> *Why?*
> 1. Let's say you review the PIP and find 10 issues. Once you quote and
> comment on those ten points, you basically started 10 threads of
> conversations.
> After 2-3 ping pongs with quotes of quotes of quotes, it takes you forever
> to read each thread properly. You find your CTRL-F to search to find your
> original quote, and reply. Load it up again in your head, switching to the
> PIP tab to read it again.
> After 10 ping pongs, it becomes almost an impossible mission.
>
> I can say I'm 75% tired just from the process, not from the review itself.
>
> 2. It's non collaborative by design.
> After 10 ping pongs, the ability of someone to come and join the discussion
> is 0. They need to go through so many replies, which are half quotes, find
> the original reply, and browse to the PIP.
> It's no wonder people drop off the PIP review once we cross 5-6 replies.
> It's no wonder, nobody joins after 10 replies.
>
> 3. It's not open to the public. Non collaborative.
> You can't just get a link, and join the review. Not only because of (1) and
> (2). You need to join the mailing list. You don't get the past emails to
> reply. Just joining the list is a high enough bar for many people.
> I personally participated in review of proposals in OpenTelemetry in the
> last 6 months, even though I'm just an occasional user.  Why? They were
> conducted on GitHub PR , so it was easy for me - click a link and reply.
>
> 4. All over the place
> Sometimes people comment on the GitHub issue.
> Sometimes on the mailing list.
> Not a single place.
>
> 5. No history.
> Ok, finally the author was convinced. I can't see just the changes. They
> need to explicitly tell me something was changed.
>
> 6. Delete All.
> They can go crazy, after 1 year, edit the GitHub issue , delete all the
> text and write "Kafka is the king". No history, nobody can stop them. It's
> their issue.
>
> 7. Show me all the approved PIPs
> Hard to track it today, hard to maintain it updated.
>
> 8. Resolved comments
> Even though you managed to read all 35 replies so far, in reply 36 you see
> the author agreed to all 8 out of 10 suggestions. You have no idea of
> knowing that in advance. You just wasted 1 hour.
>
>
> *What do I suggest?*
>
> PR is the main tool we have that allows multiple threaded discussion.
> Git provides history. You can't delete it without approval from PMC members.
>
> 1. We'll create a folder named "pip" in the pulsar main repo. It will
> contain one markdown file for each PIP. The file will be named
> "pip-xxx.md".I will write below how to obtain XXX before you start.
> 2. To create a PIP, you grab "pip/template.md" and use it to compose your
> file in the pip folder.
> 3. You submit this file as a PR named "PIP-xxx: short description".
> 4. You create "[DISCUSS] PIP-xxx: short description" in the DEV mailing
> list and refer people to your PR, with short text explaining the gist of it.
> 5. People discuss using PR comments, each is its own threaded comment.
> 6. Comment was done discussion? They resolve it. This way you see what the
> pending discussions are at a glance.
> 7. Reached consensus? Good. Send "[VOTE] PIP-xxx: short description" on DEV
> mailing list.
> 8. PIP approved? Awesome. Push commit with link to vote.
> A PMC member will merge it.
> Merge == approved.
> PMC members can add a PIP label.
> 9. Rejected? All good. Close the PR.
>  Closed == Rejected.
>  It can't be deleted. All comments are still here.
>
> Before you start, you search Pull Requests with label PIP in GitHub (`is:pr
> "PIP-" in:title`)
> Take the biggest number and add 1.
> It is super rare to have two people create PR at the same time.
>
> *Show me all approved PIPs:*
> Search merged PRs labeled PIP.
> Look at "pip" folder
>
> *Show me rejected PIPs:*
> Search closed PRs with "PIP-" in title, or labeled PIP.
>
>
> This is very similar to how Apache BK works.
>
> WDYT?


Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread tison
Hi Asaf,

Thanks for starting this thread!

I have similar thoughts on using a single source for reviewing PIPs. GH PRs
are good for conversation, although multiple conversations are still hard
to follow (which can be natural)

Here is how Rust does it[1] - a self-documented RFC repo + review PRs +
tracking issue on the main repo once accepted.

Here is what I designed for TiDB[2][3]; proposals are placed in a folder
under the main repo.

We don't have to follow other community's ways, but these two practices
seem good to read.

And, to follow the ASF voting strategy[4][5], we may still need a standard
vote mail thread and document the result on the mailing list.

Best,
tison.

[1] https://github.com/rust-lang/rfcs
[2] https://github.com/pingcap/tidb/tree/master/docs/design
[3]
https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/make-a-proposal.html
[4] https://www.apache.org/foundation/voting.html
[5] https://www.apache.org/dev/pmc.html#mailing-list-naming-policy


Girish Sharma  于2023年3月31日周五 04:33写道:

> +1 (non-binding .. ? )
> I've already commented a couple of times (here and there) that the process
> needs to be consolidated at a single place. This is a good and detailed
> approach.
> Not sure if there is a historical context behind keeping the discussion in
> dev mailing list..
>
> Regards
>
> On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika  wrote:
>
> > Hi all,
> >
> > In the last 2 months, I've increased my PIP review time (I do it in
> > cycles), and reviewed quite a few PIPs.
> >
> > My conclusion as a result of that:
> >
> > It's nearly impossible to review PIPs using a mailing list.
> > We must fix it soon.
> >
> > *Why?*
> > 1. Let's say you review the PIP and find 10 issues. Once you quote and
> > comment on those ten points, you basically started 10 threads of
> > conversations.
> > After 2-3 ping pongs with quotes of quotes of quotes, it takes you
> forever
> > to read each thread properly. You find your CTRL-F to search to find your
> > original quote, and reply. Load it up again in your head, switching to
> the
> > PIP tab to read it again.
> > After 10 ping pongs, it becomes almost an impossible mission.
> >
> > I can say I'm 75% tired just from the process, not from the review
> itself.
> >
> > 2. It's non collaborative by design.
> > After 10 ping pongs, the ability of someone to come and join the
> discussion
> > is 0. They need to go through so many replies, which are half quotes,
> find
> > the original reply, and browse to the PIP.
> > It's no wonder people drop off the PIP review once we cross 5-6 replies.
> > It's no wonder, nobody joins after 10 replies.
> >
> > 3. It's not open to the public. Non collaborative.
> > You can't just get a link, and join the review. Not only because of (1)
> and
> > (2). You need to join the mailing list. You don't get the past emails to
> > reply. Just joining the list is a high enough bar for many people.
> > I personally participated in review of proposals in OpenTelemetry in the
> > last 6 months, even though I'm just an occasional user.  Why? They were
> > conducted on GitHub PR , so it was easy for me - click a link and reply.
> >
> > 4. All over the place
> > Sometimes people comment on the GitHub issue.
> > Sometimes on the mailing list.
> > Not a single place.
> >
> > 5. No history.
> > Ok, finally the author was convinced. I can't see just the changes. They
> > need to explicitly tell me something was changed.
> >
> > 6. Delete All.
> > They can go crazy, after 1 year, edit the GitHub issue , delete all the
> > text and write "Kafka is the king". No history, nobody can stop them.
> It's
> > their issue.
> >
> > 7. Show me all the approved PIPs
> > Hard to track it today, hard to maintain it updated.
> >
> > 8. Resolved comments
> > Even though you managed to read all 35 replies so far, in reply 36 you
> see
> > the author agreed to all 8 out of 10 suggestions. You have no idea of
> > knowing that in advance. You just wasted 1 hour.
> >
> >
> > *What do I suggest?*
> >
> > PR is the main tool we have that allows multiple threaded discussion.
> > Git provides history. You can't delete it without approval from PMC
> > members.
> >
> > 1. We'll create a folder named "pip" in the pulsar main repo. It will
> > contain one markdown file for each PIP. The file will be named
> > "pip-xxx.md".I will write below how to obtain XXX before you start.
> > 2. To create a PIP, you grab "pip/template.md" and use it to compose your
> > file in the pip folder.
> > 3. You submit this file as a PR named "PIP-xxx: short description".
> > 4. You create "[DISCUSS] PIP-xxx: short description" in the DEV mailing
> > list and refer people to your PR, with short text explaining the gist of
> > it.
> > 5. People discuss using PR comments, each is its own threaded comment.
> > 6. Comment was done discussion? They resolve it. This way you see what
> the
> > pending discussions are at a glance.
> > 7. Reached consensus? Good. Send "[VOTE] PIP-xxx: short 

[DISCUSS] Add checklist for PMC binding vote of PIP

2023-03-30 Thread Asaf Mesika
Hi,

When you read last year's PIPs, many lack background information, hard to
read and understand even if you know pulsar in and out.

First step to fix was to change the PIP is structured:
https://github.com/apache/pulsar/pull/19832

In my opinion, when someone votes "+1" and it's binding, they basically
take the responsibility to say:

* I read the PIP fully.
* A person having basic Pulsar user knowledge, can read the PIP and fully
understand it
  Why? Since it contains all background information necessary to
understand the problem and the solution
   It is written in a coherent and easy to understand way.
* I validated the solution technically and can vouch for it.
   Examples:
   The PIP adds schema compatibility rules for Protobuf Native.
 I learned / know protobuf well.
 I validated the rules written containing all rules needed and
not containing wrong rules, or missing rules.

   The PIP adds new OpenID Connect authentication.
  I learned / know Authentication in Pulsar.
   I learned / know OpenID connect
   I validated the solution is architecturally correct and
sound.

Basically the PMC member voting +1 on it, basically acts as Tech Lead of
Pulsar for this PIP.
It's a very big responsibility.
It's the only way to ensure Pulsar architecture won't go haywire over the
next few years.

Yes, it will slow the process down.
Yes, it will be harder to find people to review it like that.

But, it will raise the bar for PIPs and for Pulsar architecture overall.
IMO we need that, and it's customary.

*My suggestion*
When PMC member replies to vote, it will look like this:

"
+1 (binding)

[v] PIP has all sections detailed in the PIP template (Background,
motivation, etc.)
[v] A person having basic Pulsar user knowledge, can read the PIP and fully
understand it
[v] I read PIP and validated it technically
"

or
"
-1 (binding)

I think this PIP needs:
...
"

Thanks,

Asaf


Re: [DISCUSS] We must change the way we review PIPs

2023-03-30 Thread Girish Sharma
+1 (non-binding .. ? )
I've already commented a couple of times (here and there) that the process
needs to be consolidated at a single place. This is a good and detailed
approach.
Not sure if there is a historical context behind keeping the discussion in
dev mailing list..

Regards

On Fri, Mar 31, 2023 at 1:57 AM Asaf Mesika  wrote:

> Hi all,
>
> In the last 2 months, I've increased my PIP review time (I do it in
> cycles), and reviewed quite a few PIPs.
>
> My conclusion as a result of that:
>
> It's nearly impossible to review PIPs using a mailing list.
> We must fix it soon.
>
> *Why?*
> 1. Let's say you review the PIP and find 10 issues. Once you quote and
> comment on those ten points, you basically started 10 threads of
> conversations.
> After 2-3 ping pongs with quotes of quotes of quotes, it takes you forever
> to read each thread properly. You find your CTRL-F to search to find your
> original quote, and reply. Load it up again in your head, switching to the
> PIP tab to read it again.
> After 10 ping pongs, it becomes almost an impossible mission.
>
> I can say I'm 75% tired just from the process, not from the review itself.
>
> 2. It's non collaborative by design.
> After 10 ping pongs, the ability of someone to come and join the discussion
> is 0. They need to go through so many replies, which are half quotes, find
> the original reply, and browse to the PIP.
> It's no wonder people drop off the PIP review once we cross 5-6 replies.
> It's no wonder, nobody joins after 10 replies.
>
> 3. It's not open to the public. Non collaborative.
> You can't just get a link, and join the review. Not only because of (1) and
> (2). You need to join the mailing list. You don't get the past emails to
> reply. Just joining the list is a high enough bar for many people.
> I personally participated in review of proposals in OpenTelemetry in the
> last 6 months, even though I'm just an occasional user.  Why? They were
> conducted on GitHub PR , so it was easy for me - click a link and reply.
>
> 4. All over the place
> Sometimes people comment on the GitHub issue.
> Sometimes on the mailing list.
> Not a single place.
>
> 5. No history.
> Ok, finally the author was convinced. I can't see just the changes. They
> need to explicitly tell me something was changed.
>
> 6. Delete All.
> They can go crazy, after 1 year, edit the GitHub issue , delete all the
> text and write "Kafka is the king". No history, nobody can stop them. It's
> their issue.
>
> 7. Show me all the approved PIPs
> Hard to track it today, hard to maintain it updated.
>
> 8. Resolved comments
> Even though you managed to read all 35 replies so far, in reply 36 you see
> the author agreed to all 8 out of 10 suggestions. You have no idea of
> knowing that in advance. You just wasted 1 hour.
>
>
> *What do I suggest?*
>
> PR is the main tool we have that allows multiple threaded discussion.
> Git provides history. You can't delete it without approval from PMC
> members.
>
> 1. We'll create a folder named "pip" in the pulsar main repo. It will
> contain one markdown file for each PIP. The file will be named
> "pip-xxx.md".I will write below how to obtain XXX before you start.
> 2. To create a PIP, you grab "pip/template.md" and use it to compose your
> file in the pip folder.
> 3. You submit this file as a PR named "PIP-xxx: short description".
> 4. You create "[DISCUSS] PIP-xxx: short description" in the DEV mailing
> list and refer people to your PR, with short text explaining the gist of
> it.
> 5. People discuss using PR comments, each is its own threaded comment.
> 6. Comment was done discussion? They resolve it. This way you see what the
> pending discussions are at a glance.
> 7. Reached consensus? Good. Send "[VOTE] PIP-xxx: short description" on DEV
> mailing list.
> 8. PIP approved? Awesome. Push commit with link to vote.
> A PMC member will merge it.
> Merge == approved.
> PMC members can add a PIP label.
> 9. Rejected? All good. Close the PR.
>  Closed == Rejected.
>  It can't be deleted. All comments are still here.
>
> Before you start, you search Pull Requests with label PIP in GitHub (`is:pr
> "PIP-" in:title`)
> Take the biggest number and add 1.
> It is super rare to have two people create PR at the same time.
>
> *Show me all approved PIPs:*
> Search merged PRs labeled PIP.
> Look at "pip" folder
>
> *Show me rejected PIPs:*
> Search closed PRs with "PIP-" in title, or labeled PIP.
>
>
> This is very similar to how Apache BK works.
>
> WDYT?
>


-- 
Girish Sharma


[DISCUSS] We must change the way we review PIPs

2023-03-30 Thread Asaf Mesika
Hi all,

In the last 2 months, I've increased my PIP review time (I do it in
cycles), and reviewed quite a few PIPs.

My conclusion as a result of that:

It's nearly impossible to review PIPs using a mailing list.
We must fix it soon.

*Why?*
1. Let's say you review the PIP and find 10 issues. Once you quote and
comment on those ten points, you basically started 10 threads of
conversations.
After 2-3 ping pongs with quotes of quotes of quotes, it takes you forever
to read each thread properly. You find your CTRL-F to search to find your
original quote, and reply. Load it up again in your head, switching to the
PIP tab to read it again.
After 10 ping pongs, it becomes almost an impossible mission.

I can say I'm 75% tired just from the process, not from the review itself.

2. It's non collaborative by design.
After 10 ping pongs, the ability of someone to come and join the discussion
is 0. They need to go through so many replies, which are half quotes, find
the original reply, and browse to the PIP.
It's no wonder people drop off the PIP review once we cross 5-6 replies.
It's no wonder, nobody joins after 10 replies.

3. It's not open to the public. Non collaborative.
You can't just get a link, and join the review. Not only because of (1) and
(2). You need to join the mailing list. You don't get the past emails to
reply. Just joining the list is a high enough bar for many people.
I personally participated in review of proposals in OpenTelemetry in the
last 6 months, even though I'm just an occasional user.  Why? They were
conducted on GitHub PR , so it was easy for me - click a link and reply.

4. All over the place
Sometimes people comment on the GitHub issue.
Sometimes on the mailing list.
Not a single place.

5. No history.
Ok, finally the author was convinced. I can't see just the changes. They
need to explicitly tell me something was changed.

6. Delete All.
They can go crazy, after 1 year, edit the GitHub issue , delete all the
text and write "Kafka is the king". No history, nobody can stop them. It's
their issue.

7. Show me all the approved PIPs
Hard to track it today, hard to maintain it updated.

8. Resolved comments
Even though you managed to read all 35 replies so far, in reply 36 you see
the author agreed to all 8 out of 10 suggestions. You have no idea of
knowing that in advance. You just wasted 1 hour.


*What do I suggest?*

PR is the main tool we have that allows multiple threaded discussion.
Git provides history. You can't delete it without approval from PMC members.

1. We'll create a folder named "pip" in the pulsar main repo. It will
contain one markdown file for each PIP. The file will be named
"pip-xxx.md".I will write below how to obtain XXX before you start.
2. To create a PIP, you grab "pip/template.md" and use it to compose your
file in the pip folder.
3. You submit this file as a PR named "PIP-xxx: short description".
4. You create "[DISCUSS] PIP-xxx: short description" in the DEV mailing
list and refer people to your PR, with short text explaining the gist of it.
5. People discuss using PR comments, each is its own threaded comment.
6. Comment was done discussion? They resolve it. This way you see what the
pending discussions are at a glance.
7. Reached consensus? Good. Send "[VOTE] PIP-xxx: short description" on DEV
mailing list.
8. PIP approved? Awesome. Push commit with link to vote.
A PMC member will merge it.
Merge == approved.
PMC members can add a PIP label.
9. Rejected? All good. Close the PR.
 Closed == Rejected.
 It can't be deleted. All comments are still here.

Before you start, you search Pull Requests with label PIP in GitHub (`is:pr
"PIP-" in:title`)
Take the biggest number and add 1.
It is super rare to have two people create PR at the same time.

*Show me all approved PIPs:*
Search merged PRs labeled PIP.
Look at "pip" folder

*Show me rejected PIPs:*
Search closed PRs with "PIP-" in title, or labeled PIP.


This is very similar to how Apache BK works.

WDYT?


Re: [Python] Should we make the schema default compatible with Java client?

2023-03-30 Thread Yunze Xu
> Will it register a new schema?

Only when it could pass the schema compatibility strategy. BTW, the
existing schema compatibility checker does not check the order of
fields, while it is very important. IMO, it's a bug of the broker.

Just checked this thread and found I didn't paste this issue:
https://github.com/apache/pulsar-client-python/issues/108. You can see
the schema compatibility strategy is FORWARD, then the sorted schema
from the Java client overwrote the unsorted schema from the Python
client. However, the Python consumer that uses the old schema failed
to decode the message of the new schema.

My goal is to make the Python client act the same as the Java client
since the next formal release. Regarding how the broker processes it,
I think it's another thing to be fixed.

Thanks,
Yunze

On Thu, Mar 30, 2023 at 8:42 PM 丛搏  wrote:
>
> Hi, Yunze:
>
> > Regarding the 1st question, yes, that's why I open this thread to
> > discuss. If we change these default values, the behavior of new Python
> > clients will be like the Java client. In addition, it actually reverts
> > the breaking change brought in #12232.
>
> I also kind of forget why we have #12232 to change the default behavior
> Maybe the python2 and python3 order rule is different.
>
> If we change the order is the default value, for every topic that uses
> python client will register a new schema. Will it register a new
> schema? Maybe we should add a special logic in the broker to
> check the python client version and make it will not register
> a new schema. Otherwise, the impact may still be quite large.
>
> Thanks,
> Bo
> >
> > Regarding the 2nd question, yes, they are both sorted in alphabetical
> > order. I don't know the behavior of the .NET clients, for C++, Golang,
> > Node.js clients, they all do not support generating schema definition
> > from a DTO.
> >
> > Thanks,
> > Yunze
> >
> > On Thu, Mar 30, 2023 at 10:14 AM 丛搏  wrote:
> > >
> > > Hi, Yunze :
> > >
> > > 1. If the changes may cause some compatibility issues.
> > > How do we solve the compatibility issues? It may be a
> > > breaking change.
> > >
> > > 2. Another question is if sorting is enabled by default,
> > > is the sorting rule the same as java or other clients?
> > >
> > > Putting aside the above two problems, I think it is
> > > good to be consistent with other clients.
> > >
> > > Thanks,
> > > Bo
> > >
> > > Eric Hare  于2023年3月29日周三 22:42写道:
> > > >
> > > > +1 - i think keeping the `_sorted_fields` and `_required` defaults 
> > > > consistent between the clients is the way to go.
> > > >
> > > > > On Mar 29, 2023, at 7:09 AM, Yunze Xu  
> > > > > wrote:
> > > > >
> > > > > I found the Python client has two options to control the behavior:
> > > > > 1. Set `_sorted_fields`. It's false by default in the Python client,
> > > > > but it's true in the Java client. i.e. the Java client sorts all
> > > > > fields by default.
> > > > > 2. Set `_required`. It's false by default for all types in the Python
> > > > > client, but it's only false for the string type in the Java client.
> > > > >
> > > > > i.e. given the following Java class:
> > > > >
> > > > > ```java
> > > > > class User {
> > > > >String name;
> > > > >int age;
> > > > >double score;
> > > > > }
> > > > > ```
> > > > >
> > > > > We have to give the following definition in Python:
> > > > >
> > > > > ```python
> > > > > class User(Record):
> > > > >_sorted_fields = True
> > > > >name = String()
> > > > >age = Integer(required=True)
> > > > >score = Double(required=True)
> > > > > ```
> > > > >
> > > > > I see https://github.com/apache/pulsar/pull/12232 adds the
> > > > > `_sorted_fields` field and disables the field sort by default. It
> > > > > breaks compatibility with the Java client.
> > > > >
> > > > > IMO, we should make `_sorted_fields` true by default and `_required`
> > > > > true for all types other than `String` by default.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Wed, Mar 29, 2023 at 4:00 PM Yunze Xu  wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> Recently I found the default generated schema definition in the 
> > > > >> Python
> > > > >> client is different from the Java client, which leads to some
> > > > >> unexpected behavior.
> > > > >>
> > > > >> For example, given the following class definition in Python:
> > > > >>
> > > > >> ```python
> > > > >> class Data(Record):
> > > > >>i = Integer()
> > > > >> ```
> > > > >>
> > > > >> The type of `i` field is a union: "type": ["null", "int"]
> > > > >>
> > > > >> While given the following class definition in Java:
> > > > >>
> > > > >> ```java
> > > > >> class Data {
> > > > >>private final int i;
> > > > >>/* ... */
> > > > >> }
> > > > >> ```
> > > > >>
> > > > >> The type of `i` field is an integer: "type": "int"
> > > > >>
> > > > >> It brings an issue that if a Python consumer subscribes to a topic
> > > > >> with schema defined above, then a Java producer will 

Re: [ANNOUNCE] Qiang Zhao as new PMC member in Apache Pulsar

2023-03-30 Thread Zixuan Liu
Congrats!

Thanks,
Zixuan

太上玄元道君  于2023年3月30日周四 01:40写道:

> Congrats!!
>
> Thanks,
> Tao Jiuming
>
> > 2023年3月29日 23:51,Devin Bost  写道:
> >
> > Congrats!
> >
> > Devin G. Bost
> >
> >
> > On Wed, Mar 29, 2023 at 6:38 AM ZhangJian He  wrote:
> >
> >> Congratulations!
> >>
> >> Thanks
> >> ZhangJian He
> >>
> >>
> >> On Wed, 29 Mar 2023 at 19:33, Haiting Jiang 
> >> wrote:
> >>
> >>> Congratulations!
> >>>
> >>>
> >>> Haiting
> >>>
> >>> On Wed, Mar 29, 2023 at 5:29 PM Cong Zhao  wrote:
> 
>  Congrats! Qiang.
> 
> 
>  Thanks,
>  Cong Zhao
> 
>  On 2023/03/29 03:22:43 guo jiwei wrote:
> > Dear Community,
> >
> > We are thrilled to announce that Qiang Zhao
> > (https://github.com/mattisonchao) has been invited and has accepted
> >>> the
> > role of member of the Apache Pulsar Project Management Committee
> >> (PMC).
> >
> > Qiang has been a vital asset to our community, consistently
> > demonstrating his dedication and active participation through
> > significant contributions. In addition to his technical
> >> contributions,
> > Qiang also plays an important role in reviewing pull requests and
> > ensuring the overall quality of our project. We look forward to his
> > continued contributions.
> >
> > On behalf of the Pulsar PMC, we extend a warm welcome and
> > congratulations to Qiang Zhao.
> >
> > Best regards
> > Jiwei
> >
> >>>
> >>
>
>


Re: [Python] Should we make the schema default compatible with Java client?

2023-03-30 Thread 丛搏
Hi, Yunze:

> Regarding the 1st question, yes, that's why I open this thread to
> discuss. If we change these default values, the behavior of new Python
> clients will be like the Java client. In addition, it actually reverts
> the breaking change brought in #12232.

I also kind of forget why we have #12232 to change the default behavior
Maybe the python2 and python3 order rule is different.

If we change the order is the default value, for every topic that uses
python client will register a new schema. Will it register a new
schema? Maybe we should add a special logic in the broker to
check the python client version and make it will not register
a new schema. Otherwise, the impact may still be quite large.

Thanks,
Bo
>
> Regarding the 2nd question, yes, they are both sorted in alphabetical
> order. I don't know the behavior of the .NET clients, for C++, Golang,
> Node.js clients, they all do not support generating schema definition
> from a DTO.
>
> Thanks,
> Yunze
>
> On Thu, Mar 30, 2023 at 10:14 AM 丛搏  wrote:
> >
> > Hi, Yunze :
> >
> > 1. If the changes may cause some compatibility issues.
> > How do we solve the compatibility issues? It may be a
> > breaking change.
> >
> > 2. Another question is if sorting is enabled by default,
> > is the sorting rule the same as java or other clients?
> >
> > Putting aside the above two problems, I think it is
> > good to be consistent with other clients.
> >
> > Thanks,
> > Bo
> >
> > Eric Hare  于2023年3月29日周三 22:42写道:
> > >
> > > +1 - i think keeping the `_sorted_fields` and `_required` defaults 
> > > consistent between the clients is the way to go.
> > >
> > > > On Mar 29, 2023, at 7:09 AM, Yunze Xu  
> > > > wrote:
> > > >
> > > > I found the Python client has two options to control the behavior:
> > > > 1. Set `_sorted_fields`. It's false by default in the Python client,
> > > > but it's true in the Java client. i.e. the Java client sorts all
> > > > fields by default.
> > > > 2. Set `_required`. It's false by default for all types in the Python
> > > > client, but it's only false for the string type in the Java client.
> > > >
> > > > i.e. given the following Java class:
> > > >
> > > > ```java
> > > > class User {
> > > >String name;
> > > >int age;
> > > >double score;
> > > > }
> > > > ```
> > > >
> > > > We have to give the following definition in Python:
> > > >
> > > > ```python
> > > > class User(Record):
> > > >_sorted_fields = True
> > > >name = String()
> > > >age = Integer(required=True)
> > > >score = Double(required=True)
> > > > ```
> > > >
> > > > I see https://github.com/apache/pulsar/pull/12232 adds the
> > > > `_sorted_fields` field and disables the field sort by default. It
> > > > breaks compatibility with the Java client.
> > > >
> > > > IMO, we should make `_sorted_fields` true by default and `_required`
> > > > true for all types other than `String` by default.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Wed, Mar 29, 2023 at 4:00 PM Yunze Xu  wrote:
> > > >>
> > > >> Hi all,
> > > >>
> > > >> Recently I found the default generated schema definition in the Python
> > > >> client is different from the Java client, which leads to some
> > > >> unexpected behavior.
> > > >>
> > > >> For example, given the following class definition in Python:
> > > >>
> > > >> ```python
> > > >> class Data(Record):
> > > >>i = Integer()
> > > >> ```
> > > >>
> > > >> The type of `i` field is a union: "type": ["null", "int"]
> > > >>
> > > >> While given the following class definition in Java:
> > > >>
> > > >> ```java
> > > >> class Data {
> > > >>private final int i;
> > > >>/* ... */
> > > >> }
> > > >> ```
> > > >>
> > > >> The type of `i` field is an integer: "type": "int"
> > > >>
> > > >> It brings an issue that if a Python consumer subscribes to a topic
> > > >> with schema defined above, then a Java producer will fail to create
> > > >> because of the schema incompatibility.
> > > >>
> > > >> Currently, the workaround is to change the schema compatibility
> > > >> strategy to FORWARD.
> > > >>
> > > >> Should we change the way to generate schema definition in the Python
> > > >> client to be compatible with the Java client? It could bring breaking
> > > >> changes to old Python clients, but it could guarantee compatibility
> > > >> with the Java client.
> > > >>
> > > >> If not, we still have to introduce an extra configuration to make
> > > >> Python schema compatible with Java schema. But it requires code
> > > >> changes. e.g. here is a possible solution:
> > > >>
> > > >> ```python
> > > >> class Data(Record):
> > > >># NOTE: Users might have to add this extra field to control how to
> > > >> generate the schema
> > > >>__java_compatible = True
> > > >>i = Integer()
> > > >> ```
> > > >>
> > > >> Thanks,
> > > >> Yunze
> > >


Re: [Python] Should we make the schema default compatible with Java client?

2023-03-30 Thread Yunze Xu
Hi Bo,

Regarding the 1st question, yes, that's why I open this thread to
discuss. If we change these default values, the behavior of new Python
clients will be like the Java client. In addition, it actually reverts
the breaking change brought in #12232.

Regarding the 2nd question, yes, they are both sorted in alphabetical
order. I don't know the behavior of the .NET clients, for C++, Golang,
Node.js clients, they all do not support generating schema definition
from a DTO.

Thanks,
Yunze

On Thu, Mar 30, 2023 at 10:14 AM 丛搏  wrote:
>
> Hi, Yunze :
>
> 1. If the changes may cause some compatibility issues.
> How do we solve the compatibility issues? It may be a
> breaking change.
>
> 2. Another question is if sorting is enabled by default,
> is the sorting rule the same as java or other clients?
>
> Putting aside the above two problems, I think it is
> good to be consistent with other clients.
>
> Thanks,
> Bo
>
> Eric Hare  于2023年3月29日周三 22:42写道:
> >
> > +1 - i think keeping the `_sorted_fields` and `_required` defaults 
> > consistent between the clients is the way to go.
> >
> > > On Mar 29, 2023, at 7:09 AM, Yunze Xu  
> > > wrote:
> > >
> > > I found the Python client has two options to control the behavior:
> > > 1. Set `_sorted_fields`. It's false by default in the Python client,
> > > but it's true in the Java client. i.e. the Java client sorts all
> > > fields by default.
> > > 2. Set `_required`. It's false by default for all types in the Python
> > > client, but it's only false for the string type in the Java client.
> > >
> > > i.e. given the following Java class:
> > >
> > > ```java
> > > class User {
> > >String name;
> > >int age;
> > >double score;
> > > }
> > > ```
> > >
> > > We have to give the following definition in Python:
> > >
> > > ```python
> > > class User(Record):
> > >_sorted_fields = True
> > >name = String()
> > >age = Integer(required=True)
> > >score = Double(required=True)
> > > ```
> > >
> > > I see https://github.com/apache/pulsar/pull/12232 adds the
> > > `_sorted_fields` field and disables the field sort by default. It
> > > breaks compatibility with the Java client.
> > >
> > > IMO, we should make `_sorted_fields` true by default and `_required`
> > > true for all types other than `String` by default.
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Wed, Mar 29, 2023 at 4:00 PM Yunze Xu  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> Recently I found the default generated schema definition in the Python
> > >> client is different from the Java client, which leads to some
> > >> unexpected behavior.
> > >>
> > >> For example, given the following class definition in Python:
> > >>
> > >> ```python
> > >> class Data(Record):
> > >>i = Integer()
> > >> ```
> > >>
> > >> The type of `i` field is a union: "type": ["null", "int"]
> > >>
> > >> While given the following class definition in Java:
> > >>
> > >> ```java
> > >> class Data {
> > >>private final int i;
> > >>/* ... */
> > >> }
> > >> ```
> > >>
> > >> The type of `i` field is an integer: "type": "int"
> > >>
> > >> It brings an issue that if a Python consumer subscribes to a topic
> > >> with schema defined above, then a Java producer will fail to create
> > >> because of the schema incompatibility.
> > >>
> > >> Currently, the workaround is to change the schema compatibility
> > >> strategy to FORWARD.
> > >>
> > >> Should we change the way to generate schema definition in the Python
> > >> client to be compatible with the Java client? It could bring breaking
> > >> changes to old Python clients, but it could guarantee compatibility
> > >> with the Java client.
> > >>
> > >> If not, we still have to introduce an extra configuration to make
> > >> Python schema compatible with Java schema. But it requires code
> > >> changes. e.g. here is a possible solution:
> > >>
> > >> ```python
> > >> class Data(Record):
> > >># NOTE: Users might have to add this extra field to control how to
> > >> generate the schema
> > >>__java_compatible = True
> > >>i = Integer()
> > >> ```
> > >>
> > >> Thanks,
> > >> Yunze
> >


Re: [VOTE] Pulsar Client Go Release 0.10.0 Candidate 1

2023-03-30 Thread Takeshi Kimura
+1 (non-binding)

- verified checksum and signature
- ran producer and consumer examples

Regards,
Takeshi Kimura

-元のメッセージ-
送信元: Zike Yang 
Reply-To: "dev@pulsar.apache.org" 
日付: 2023年3月27日 月曜日 21:23
宛先: "dev@pulsar.apache.org" 
件名: [VOTE] Pulsar Client Go Release 0.10.0 Candidate 1

Hi everyone,
Please review and vote on the release candidate #1 for the version
0.10.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

This is the first release candidate for Apache Pulsar Go client, version 
0.10.0.

It fixes the following issues:
https://github.com/apache/pulsar-client-go/milestone/12?closed=1

Pulsar Client Go's KEYS file contains PGP keys we used to sign this release:

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS=05%7C01%7Ctakeshki%40yahoo-corp.jp%7C7fa931e704744220829608db2ebe1fcb%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638155166332495746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=sriR%2Bg7eOnCvQ8S5iKk26fND24GXgZXmXDbxcuYMieI%3D=0

Please download these packages and review this release candidate:
- Review release notes: https://github.com/apache/pulsar-client-go/pull/997
- Download the source package (verify shasum, and asc) and follow the
README.md to build and run the pulsar-client-go.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Source file:

https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-client-go-0.10.0-candidate-1%2F=05%7C01%7Ctakeshki%40yahoo-corp.jp%7C7fa931e704744220829608db2ebe1fcb%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C638155166332495746%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=%2F64IHSR%2BcwQZXxJ%2BMsEag9DmMG0lt3IzHBLCLCnM150%3D=0

The tag to be voted upon:
v0.10.0
https://github.com/apache/pulsar-client-go/tree/v0.10.0-candidate-1

SHA-512 checksums:

5f26d95061eb535595043ffc23424d9913f11d1b80ca815bdd20577453f6e08c2a5dd8b729d82494c4f5d0260905b231218c6a437995d893c4c25efdb0bb
 apache-pulsar-client-go-0.10.0-src.tar.gz

Zike Yang



Re: [VOTE] Pulsar Client Go Release 0.10.0 Candidate 1

2023-03-30 Thread Yunze Xu
+1 (binding)

- Checked the signature and checksum
- Ran basic end-to-end tests
- Ran pulsar-perf with batch index ACK enabled

Thanks,
Yunze

On Thu, Mar 30, 2023 at 3:17 PM Nozomi Kurihara  wrote:
>
> +1 (binding)
>
> - verified checksum and signature
> - ran producer and consumer in examples
>
> Thanks,
> Nozomi
>
>
> 2023年3月28日(火) 0:04 Baodi Shi :
>
> >  +1(non-binding)
> >
> > - Checked the signature
> > - Verify producer, consumer, and reader examples on README.
> >
> > Thanks,
> > Baodi Shi
> >
> >
> > 在 2023年3月27日 20:23:28 上,Zike Yang  写道:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> > > 0.10.0, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > This is the first release candidate for Apache Pulsar Go client, version
> > > 0.10.0.
> > >
> > > It fixes the following issues:
> > > https://github.com/apache/pulsar-client-go/milestone/12?closed=1
> > >
> > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this
> > > release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > > - Review release notes:
> > > https://github.com/apache/pulsar-client-go/pull/997
> > > - Download the source package (verify shasum, and asc) and follow the
> > > README.md to build and run the pulsar-client-go.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Source file:
> > >
> > >
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.10.0-candidate-1/
> > >
> > > The tag to be voted upon:
> > > v0.10.0
> > > https://github.com/apache/pulsar-client-go/tree/v0.10.0-candidate-1
> > >
> > > SHA-512 checksums:
> > >
> > >
> > 5f26d95061eb535595043ffc23424d9913f11d1b80ca815bdd20577453f6e08c2a5dd8b729d82494c4f5d0260905b231218c6a437995d893c4c25efdb0bb
> > > apache-pulsar-client-go-0.10.0-src.tar.gz
> > >
> > > Zike Yang
> > >
> >


Re: [VOTE] Pulsar Client Go Release 0.10.0 Candidate 1

2023-03-30 Thread Nozomi Kurihara
+1 (binding)

- verified checksum and signature
- ran producer and consumer in examples

Thanks,
Nozomi


2023年3月28日(火) 0:04 Baodi Shi :

>  +1(non-binding)
>
> - Checked the signature
> - Verify producer, consumer, and reader examples on README.
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年3月27日 20:23:28 上,Zike Yang  写道:
>
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version
> > 0.10.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This is the first release candidate for Apache Pulsar Go client, version
> > 0.10.0.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar-client-go/milestone/12?closed=1
> >
> > Pulsar Client Go's KEYS file contains PGP keys we used to sign this
> > release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download these packages and review this release candidate:
> > - Review release notes:
> > https://github.com/apache/pulsar-client-go/pull/997
> > - Download the source package (verify shasum, and asc) and follow the
> > README.md to build and run the pulsar-client-go.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Source file:
> >
> >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.10.0-candidate-1/
> >
> > The tag to be voted upon:
> > v0.10.0
> > https://github.com/apache/pulsar-client-go/tree/v0.10.0-candidate-1
> >
> > SHA-512 checksums:
> >
> >
> 5f26d95061eb535595043ffc23424d9913f11d1b80ca815bdd20577453f6e08c2a5dd8b729d82494c4f5d0260905b231218c6a437995d893c4c25efdb0bb
> > apache-pulsar-client-go-0.10.0-src.tar.gz
> >
> > Zike Yang
> >
>