Re: [VOTE] Pulsar Client Python Release 3.1.0 Candidate 3

[Yunze Xu]

Hi Zike,

I've reproduced this issue successfully with:
- Python 3.8 on Ubuntu 20.04
- Python 3.10 on macOS 12

It seems to be a serious bug and I'm going to figure out the reason ASAP.

Thanks,
Yunze

On Tue, Feb 28, 2023 at 5:17 PM Zike Yang  wrote:
>
> Hi, Yunze
>
> It raises an exception when I run the consumer example.
>
> Here are my environments:
> * macos 12.06 x86_64
> * python 3.7
>
> Here are my reproduce steps:
> * Start the pulsar standalone
> * Start the consumer example
> * Start the producer example
> * The consumer can receive messages successfully.
> * Stop the consumer, then it throws some exception:
> ```
> ➜  examples git:(main) ✗ python consumer.py
> 2023-02-28 17:11:04.742 INFO  [0x113b19600] Client:87 | Subscribing on
> Topic :my-topic
> 2023-02-28 17:11:04.742 INFO  [0x113b19600] ClientConnection:190 |
> [ -> pulsar://localhost:6650] Create ClientConnection,
> timeout=1
> 2023-02-28 17:11:04.742 INFO  [0x113b19600] ConnectionPool:97 |
> Created connection for pulsar://localhost:6650
> 2023-02-28 17:11:04.745 INFO  [0x73606000] ClientConnection:388 |
> [127.0.0.1:49258 -> 127.0.0.1:6650] Connected to broker
> 2023-02-28 17:11:04.857 INFO  [0x73606000] HandlerBase:72 |
> [persistent://public/default/my-topic, my-subscription, 0] Getting
> connection from pool
> 2023-02-28 17:11:04.874 INFO  [0x73606000] ConsumerImpl:238 |
> [persistent://public/default/my-topic, my-subscription, 0] Created
> consumer on broker [127.0.0.1:49258 -> 127.0.0.1:6650]
> Received message 'hello' id='(132,30,-1,0)'
> Received message 'hello' id='(132,31,-1,0)'
> Received message 'hello' id='(132,32,-1,0)'
> Received message 'hello' id='(132,33,-1,0)'
> Received message 'hello' id='(132,34,-1,0)'
> Received message 'hello' id='(132,35,-1,0)'
> Received message 'hello' id='(132,36,-1,0)'
> Received message 'hello' id='(132,37,-1,0)'
> Received message 'hello' id='(132,38,-1,0)'
> Received message 'hello' id='(132,39,-1,0)'
> ^CKeyboardInterrupt
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
>   File "consumer.py", line 32, in 
> msg = consumer.receive()
>   File 
> "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pulsar/__init__.py",
> line 1243, in receive
> msg = self._consumer.receive()
> SystemError
> 2023-02-28 17:11:07.367 WARN  [0x113b19600] ConsumerImpl:126 |
> [persistent://public/default/my-topic, my-subscription, 0] Destroyed
> consumer which was not properly closed
> 2023-02-28 17:11:07.367 INFO  [0x113b19600] ConsumerImpl:134 |
> [persistent://public/default/my-topic, my-subscription, 0] Closed
> consumer for race condition: 0
> libc++abi: terminating with uncaught exception of type
> std::__1::bad_weak_ptr: bad_weak_ptr
> [1]56483 abort  python consumer.py
> ```
> This causes the program to crash.
>
> I'm using the example codes here:
> https://github.com/apache/pulsar-client-python/tree/main/examples
>
> Could you take a look? Thanks.
>
> BR,
> Zike Yang
>
> On Tue, Feb 28, 2023 at 10:37 AM PengHui Li  wrote:
> >
> > +1 (binding)
> >
> > - install on macos
> > - start a standalone (latest master)
> > - run the example
> > https://github.com/apache/pulsar-client-python#running-examples
> >
> > Regards,
> > Penghui
> >
> > On Thu, Feb 23, 2023 at 11:00 PM Enrico Olivelli 
> > wrote:
> >
> > > Thank you Yunze for double checking.
> > >
> > > I don't have time to test the release, so I am voting +0 (and not -1)
> > >
> > > Enrico
> > >
> > > Il giorno gio 23 feb 2023 alle ore 12:57 Yunze Xu
> > >  ha scritto:
> > > >
> > > > Hi Enrico,
> > > >
> > > > I will test more operation systems and open a discussion soon. For
> > > > now, I just tested the TLS encryption and token authentication, and I
> > > > found this issue does not exist as expected for both Python and
> > > > Node.js clients.
> > > > - Windows: Only Python client works
> > > > - Ubuntu: Both work
> > > > - macOS: Not tested yet
> > > >
> > > > But the OAuth2 authentication case doesn't work. It's caused by
> > > > https://github.com/apache/pulsar/pull/16064 and fixed by
> > > > https://github.com/apache/pulsar-client-cpp/pull/190.
> > > >
> > > > When the protocol of the issuer URL is HTTPS:
> > > > - Before #16064, OAuth2 authentication works by skipping verifying the
> > > > peer, it's dangerous for security reasons
> > > > - After #16064 and before #190, there is no way to perform OAuth2
> > > authentication
> > > > - After #190, users can configure the `tls_trust_certs_file_path` to do
> > > that.
> > > >
> > > > So I think it should not be treated as a regression and should not
> > > > block this release. We can support automatically detecting the CA
> > > > certs for OAuth2 authentication later.
> > > >
> > > > The regression should be the case when the TLS encryption is enabled.
> > > > And as I've said at the beginning, I'm going to do more tests and open
> > > > a discussion.
> > > >
> > > > Thanks,
> 

[DISCUSS] Enables pulsar to configure the `MaxRequestHeaderSize` of the internal web service

[Yubiao Feng]

Hi community

The Client API can create topic names of infinite length. But after these
topics were created, we can not manage them by Admin API because the web
service (based by Jetty) has a request-line length limit.

I propose adding a pulsar configuration `httpMaxRequestHeaderSize` to set
this internal web service config (including web servicer and proxy service).

I have submitted a PR to do this:
https://github.com/apache/pulsar/pull/19514

Due to the previous bug of loop creation of DLQ(see
https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8), I think
customers should need this configuration to remove these topics that
shouldn't exist, so I think this change should be cherry-picked into
branches 2.9, 2.10, and 2.11


Thanks
Yubiao Feng

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

[ZhangJian He]

Why? How could springboot 2.x users accept this update? What do you think
about this?

Thanks
ZhangJian He


On Wed, 1 Mar 2023 at 12:11, Zixuan Liu  wrote:

> IMO, I think we can upgrade to jersey 3.x from jersey 2.x on the
> `pulsar-admin-client`.
>
> Thanks,
> Zixuan
>
> ZhangJian He  于2023年2月28日周二 17:44写道：
>
> > ping @Zixuan Liu @Enrico Olivelli 
> > So, I am not sure if we can update to 3.x. Maybe we will need provider
> two
> > versions of **pulsar-admin-client**?
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Mon, 27 Feb 2023 at 16:37, ZhangJian He  wrote:
> >
> > > Sorry, what I want to talk is `pulsar-admin-client`, it uses jersey
> > > client, which depends on `jakarta.ws.rs-api`
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Mon, 27 Feb 2023 at 16:27, Enrico Olivelli 
> > wrote:
> > >
> > >> Are we talking about the Server side or the Pulsar Admin Client (that
> > >> uses HTTP) ?
> > >> On the server side we don't care about "SpringBoot", we are not using
> it
> > >>
> > >> Can you please elaborate more ?
> > >> Enrico
> > >>
> > >>
> > >> Il giorno lun 27 feb 2023 alle ore 09:02 Zixuan Liu
> > >>  ha scritto:
> > >> >
> > >> > Only the `canal` module requires the spring, so don't worry.
> > >> >
> > >> > If I wrong, please let me know.
> > >> >
> > >> > Thanks,
> > >> > Zixuan
> > >> >
> > >> > ZhangJian He  于2023年2月27日周一 14:04写道：
> > >> >
> > >> > > Hi @Enrico Olivelli  @Zixuan Liu
> > >> > >
> > >> > > I'm concerned that this change may create some complex
> compatibility
> > >> issues
> > >> > > between our client and Springboot. Springboot 2.x uses the package
> > >> name
> > >> > > "javax" while Springboot 3.x uses "jakarta". I'm still not
> entirely
> > >> sure
> > >> > > about the extent of its impact.
> > >> > >
> > >> > > Thanks
> > >> > > ZhangJian He
> > >> > >
> > >> > >
> > >> > > On Mon, 27 Feb 2023 at 13:12, Zixuan Liu 
> wrote:
> > >> > >
> > >> > > > > IIUC the main pain point is for Custom Servlet developers who
> > will
> > >> > > > have to migrate to the jakarta.xxx packages
> > >> > > > is this correct ?
> > >> > > >
> > >> > > > Sure!
> > >> > > >
> > >> > > > Our web service and some http client depend on the jersey. I
> > >> checked on
> > >> > > > jetty and our project. Looks like we can only use the Jakarta
> EE9.
> > >> > > >
> > >> > > > We neet to upgrade the jetty to the 11.0.x from the 9.x and
> jersey
> > >> to
> > >> > > > 3.0.x. from 2.x.
> > >> > > >
> > >> > > > jetty 11.0.x - JakartaEE 9 - Java 11, you can see this in the
> > >> > > > https://www.eclipse.org/jetty/
> > >> > > > jersey 3.0.x - JakartaEE 9 - Java 8+(It looks like it supports
> > Java
> > >> 8,
> > >> > > not
> > >> > > > sure how to implementation), you can see
> > >> > > > https://eclipse-ee4j.github.io/jersey/ and
> > >> > > >
> > >> > > >
> > >> > >
> > >>
> >
> https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest30x/modules-and-dependencies.html#d0e560
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Zixuan
> > >> > > >
> > >> > > > Enrico Olivelli  于2023年2月26日周日 20:48写道：
> > >> > > >
> > >> > > > > +1 to upgrade.
> > >> > > > >
> > >> > > > > IIUC the main pain point is for Custom Servlet developers who
> > will
> > >> > > > > have to migrate to the jakarta.xxx packages
> > >> > > > > is this correct ?
> > >> > > > >
> > >> > > > > Enrico
> > >> > > > >
> > >> > > > > Il giorno sab 25 feb 2023 alle ore 06:46 Zixuan Liu
> > >> > > > >  ha scritto:
> > >> > > > > >
> > >> > > > > > Sounds greate that migrating to jersey 3.x from jersey 2.x.
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Zixuan
> > >> > > > > >
> > >> > > > > > ZhangJian He  于2023年2月25日周六 11:01写道：
> > >> > > > > >
> > >> > > > > > > Hello, community. I want to know Which jakartaEE version
> we
> > >> plan
> > >> > > for
> > >> > > > > pulsar
> > >> > > > > > > 3.0.0. Now we use jersey 2.x, which uses Jakarta EE8.
> > >> > > > > > >
> > >> > > > > > > Maybe we will need to update the jersey.
> > >> > > > > > > Eclipse Jersey (eclipse-ee4j.github.io)
> > >> > > > > > > 
> > >> > > > > > > 2.x is for Jakarta EE8. 3.0.x is for Jakarta EE9 and 3.1.x
> > is
> > >> for
> > >> > > > > Jakarta
> > >> > > > > > > EE10.
> > >> > > > > > >
> > >> > > > > > > Thank you for your time and consideration.
> > >> > > > > > >
> > >> > > > > > > Thanks
> > >> > > > > > > ZhangJian He
> > >> > > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> > >
> >
>

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

ping @PengHuiLi. Please let me know your thoughts on this matter.

Thanks
ZhangJian He


On Tue, 28 Feb 2023 at 20:24, Enrico Olivelli  wrote:

> Il giorno mar 28 feb 2023 alle ore 10:40 ZhangJian He
>  ha scritto:
> >
> > ping @PengHuiLi @Enrico Olivelli
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Mon, 27 Feb 2023 at 14:02, ZhangJian He  wrote:
> >>
> >> > Another idea is whether it is possible to generate clients for various
> >> languages based on the current pulsar rest swagger files
> >> Yes. But I don't think this is a suitable choice for the go client. Due
> to the existing swagger file not fully complying with the specification, it
> cannot be used directly to generate client code. Personally, I don't think
> fixing it is particularly easy and there is currently no one working on it.
> >> For the go client, I would still recommend creating a new repository
> and having everyone contribute to it from there. WDYT?
>
> I agree with this approach
>
> Enrico
>
> >>
> >> Also, I would appreciate it if others could let me know your thoughts
> on this matter. @PengHuiLi @Enrico Olivelli
> >>
> >> Thanks
> >> ZhangJian He
> >>
> >>
> >> On Sun, 26 Feb 2023 at 21:04, Guangning E  wrote:
> >>>
> >>> Another idea is whether it is possible to generate clients for various
> >>> languages based on the current pulsar rest swagger files?
> >>>
> >>> Thanks,
> >>> Guangning
> >>>
> >>> On Sat, Feb 25, 2023 at 1:44 PM Zixuan Liu  wrote:
> >>>
> >>> > > I would like to suggest creating a new repository called
> >>> > "pulsar-admin-go"
> >>> >
> >>> > +1, I agreed with you.
> >>> >
> >>> > Thanks,
> >>> > Zixuan
> >>> >
> >>> > ZhangJian He  于2023年2月25日周六 11:05写道：
> >>> >
> >>> > > Following our previous discussions via email, I would like to
> suggest
> >>> > > creating a new repository called "pulsar-admin-go". This
> repository will
> >>> > > serve as a platform for all of us to contribute our code.
> Pulsarctl[0] go
> >>> > > first.
> >>> > >
> >>> > > Please let me know your thoughts on this matter.
> >>> > >
> >>> > > [0] - https://github.com/streamnative/pulsarctl
> >>> > >
> >>> > > Thanks
> >>> > > ZhangJian He
> >>> > >
> >>> > >
> >>> > > On Fri, 17 Feb 2023 at 23:24, ZhangJian He 
> wrote:
> >>> > >
> >>> > > > Thank for StreamNative for willing to donate this project. This
> means
> >>> > we
> >>> > > > don't have to develop and maintain a set of HTTP code from
> scratch. My
> >>> > > idea
> >>> > > > aligns with Yunze's, and separating it into a standalone
> >>> > pulsar-admin-go
> >>> > > > project would be better. The **pulsarctl** repo contains
> bookkeeper
> >>> > http
> >>> > > > call too. Maybe we can have a project bookkeeper-admin-go ?（it's
> a
> >>> > liitle
> >>> > > > going off-topic ）
> >>> > > >
> >>> > > > Thanks
> >>> > > > ZhangJian He
> >>> > > >
> >>> > > >
> >>> > > > On Fri, 17 Feb 2023 at 20:29, PengHui Li <
> codelipeng...@gmail.com>
> >>> > > wrote:
> >>> > > >
> >>> > > >> Hi Yunze,
> >>> > > >>
> >>> > > >> Yes, we can split it.
> >>> > > >> Both one repo with two modules or two repos works for me.
> >>> > > >>
> >>> > > >> The pulsarctl already have the admin API and CLI.
> >>> > > >> So I think we don’t need to develop another one.
> >>> > > >>
> >>> > > >> Best,
> >>> > > >> Penghui
> >>> > > >>
> >>> > > >> > On Feb 17, 2023, at 17:44, Yunze Xu
> 
> >>> > > >> wrote:
> >>> > > >> >
> >>> > > >> > Hi PengHui,
> >>> > > >> >
> >>> > > >> > Now I changed my mind a bit. Even if the pulsarctl was
> contributed
> >>> > to
> >>> > > >> > the Apache Foundation, I think we should also avoid adding it
> as the
> >>> > > >> > dependency. What we need is an API layer but not the CLI,
> while
> >>> > > >> > pulsarctl couples the API and CLI.
> >>> > > >> >
> >>> > > >> > At the moment, my expectation is:
> >>> > > >> > 1. Use a separate repo (e.g. pulsar-admin-go) to implement
> the admin
> >>> > > >> > APIs in Golang.
> >>> > > >> > 2. Depend this new repo in pulsarctl.
> >>> > > >> >
> >>> > > >> > Then we will have three Go projects:
> >>> > > >> > - pulsar-client-go: The Pulsar Go client APIs
> >>> > > >> > - pulsar-admin-go: The Pulsar Go admin APIs
> >>> > > >> > - pulsarctl: The admin CLI tool written in Go
> >>> > > >> >
> >>> > > >> > Thanks,
> >>> > > >> > Yunze
> >>> > > >> >
> >>> > > >> > On Fri, Feb 17, 2023 at 4:22 PM PengHui Li <
> peng...@apache.org>
> >>> > > wrote:
> >>> > > >> >>
> >>> > > >> >> I checked with Sijie today.
> >>> > > >> >> StreamNative can contribute the pulsarctl project to Apache
> >>> > > Foundation.
> >>> > > >> >>
> >>> > > >> >> Regards,
> >>> > > >> >> Penghui
> >>> > > >> >>
> >>> > > >> >> On Fri, Feb 17, 2023 at 4:02 PM Enrico Olivelli <
> >>> > eolive...@gmail.com
> >>> > > >
> >>> > > >> wrote:
> >>> > > >> >>
> >>> > > >> >>> I agree to add an admin API to the go client, this would be
> very
> >>> > > >> helpful.
> >>> > > >> >>>
> >>> > > >> >>> Il giorno ven 17 feb 2023 alle ore 08:44 Zixuan Liu
> >>> > > >> >>>  ha scritto:
> >>> > > 

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

[Zixuan Liu]

IMO, I think we can upgrade to jersey 3.x from jersey 2.x on the
`pulsar-admin-client`.

Thanks,
Zixuan

ZhangJian He  于2023年2月28日周二 17:44写道：

> ping @Zixuan Liu @Enrico Olivelli 
> So, I am not sure if we can update to 3.x. Maybe we will need provider two
> versions of **pulsar-admin-client**?
>
> Thanks
> ZhangJian He
>
>
> On Mon, 27 Feb 2023 at 16:37, ZhangJian He  wrote:
>
> > Sorry, what I want to talk is `pulsar-admin-client`, it uses jersey
> > client, which depends on `jakarta.ws.rs-api`
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Mon, 27 Feb 2023 at 16:27, Enrico Olivelli 
> wrote:
> >
> >> Are we talking about the Server side or the Pulsar Admin Client (that
> >> uses HTTP) ?
> >> On the server side we don't care about "SpringBoot", we are not using it
> >>
> >> Can you please elaborate more ?
> >> Enrico
> >>
> >>
> >> Il giorno lun 27 feb 2023 alle ore 09:02 Zixuan Liu
> >>  ha scritto:
> >> >
> >> > Only the `canal` module requires the spring, so don't worry.
> >> >
> >> > If I wrong, please let me know.
> >> >
> >> > Thanks,
> >> > Zixuan
> >> >
> >> > ZhangJian He  于2023年2月27日周一 14:04写道：
> >> >
> >> > > Hi @Enrico Olivelli  @Zixuan Liu
> >> > >
> >> > > I'm concerned that this change may create some complex compatibility
> >> issues
> >> > > between our client and Springboot. Springboot 2.x uses the package
> >> name
> >> > > "javax" while Springboot 3.x uses "jakarta". I'm still not entirely
> >> sure
> >> > > about the extent of its impact.
> >> > >
> >> > > Thanks
> >> > > ZhangJian He
> >> > >
> >> > >
> >> > > On Mon, 27 Feb 2023 at 13:12, Zixuan Liu  wrote:
> >> > >
> >> > > > > IIUC the main pain point is for Custom Servlet developers who
> will
> >> > > > have to migrate to the jakarta.xxx packages
> >> > > > is this correct ?
> >> > > >
> >> > > > Sure!
> >> > > >
> >> > > > Our web service and some http client depend on the jersey. I
> >> checked on
> >> > > > jetty and our project. Looks like we can only use the Jakarta EE9.
> >> > > >
> >> > > > We neet to upgrade the jetty to the 11.0.x from the 9.x and jersey
> >> to
> >> > > > 3.0.x. from 2.x.
> >> > > >
> >> > > > jetty 11.0.x - JakartaEE 9 - Java 11, you can see this in the
> >> > > > https://www.eclipse.org/jetty/
> >> > > > jersey 3.0.x - JakartaEE 9 - Java 8+(It looks like it supports
> Java
> >> 8,
> >> > > not
> >> > > > sure how to implementation), you can see
> >> > > > https://eclipse-ee4j.github.io/jersey/ and
> >> > > >
> >> > > >
> >> > >
> >>
> https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest30x/modules-and-dependencies.html#d0e560
> >> > > >
> >> > > > Thanks,
> >> > > > Zixuan
> >> > > >
> >> > > > Enrico Olivelli  于2023年2月26日周日 20:48写道：
> >> > > >
> >> > > > > +1 to upgrade.
> >> > > > >
> >> > > > > IIUC the main pain point is for Custom Servlet developers who
> will
> >> > > > > have to migrate to the jakarta.xxx packages
> >> > > > > is this correct ?
> >> > > > >
> >> > > > > Enrico
> >> > > > >
> >> > > > > Il giorno sab 25 feb 2023 alle ore 06:46 Zixuan Liu
> >> > > > >  ha scritto:
> >> > > > > >
> >> > > > > > Sounds greate that migrating to jersey 3.x from jersey 2.x.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Zixuan
> >> > > > > >
> >> > > > > > ZhangJian He  于2023年2月25日周六 11:01写道：
> >> > > > > >
> >> > > > > > > Hello, community. I want to know Which jakartaEE version we
> >> plan
> >> > > for
> >> > > > > pulsar
> >> > > > > > > 3.0.0. Now we use jersey 2.x, which uses Jakarta EE8.
> >> > > > > > >
> >> > > > > > > Maybe we will need to update the jersey.
> >> > > > > > > Eclipse Jersey (eclipse-ee4j.github.io)
> >> > > > > > > 
> >> > > > > > > 2.x is for Jakarta EE8. 3.0.x is for Jakarta EE9 and 3.1.x
> is
> >> for
> >> > > > > Jakarta
> >> > > > > > > EE10.
> >> > > > > > >
> >> > > > > > > Thank you for your time and consideration.
> >> > > > > > >
> >> > > > > > > Thanks
> >> > > > > > > ZhangJian He
> >> > > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> >
>

Re: [DISCUSS] Support custom compressionType for pulsar functions

[Pengcheng Jiang]

Use `LZ4` as zero value seems better, I will update

Rui Fu  于2023年3月1日周三 11:12写道：

> +1, very useful, just one question: why not set the `LZ4` as the “zero”
> value instead? Like for the enum with following orders: `LZ4`, `NONE`,
> `ZLIB`, `ZSTD` and `SNAPPY`? So it will remain the backward compatibility.
>
> Best,
>
> Rui Fu
> On Mar 1, 2023 at 10:18 +0800, Neng Lu , wrote:
> > +1 for the change.
> >
> > On 2023/02/28 01:06:51 Pengcheng Jiang wrote:
> > > Hello, community:
> > >
> > > ### Motivation
> > >
> > > Currently, pulsar functions are using `LZ4` as the compression type,
> and
> > > users cannot change it, yet some users may want to custom this
> behavior.
> > >
> > > ### Modifications
> > >
> > > Add a `CompressionType` field(which is an enum) to the `ProducerSpec`
> in
> > > the `Function.proto`; this enum has six values: `NOTSET`, `NONE`,
> `LZ4`,
> > > `ZLIB`, `ZSTD` and `SNAPPY`, there is a `NOTSET` value besides of 5
> > > supported compression type, so that even users don't set the
> > > `CompressionType`, it will fallback to its "zero" value: `NOTSET`
> instead
> > > of `NONE`, and in such case, pulsar function instances will use `LZ4`
> to
> > > keep the same behavior with before.
> > >
> > > PTAL when you have time and feel free to leave any comments.
> > >
> > > Best Regards,
> > > Pengcheng Jiang
> > >
> > > [0] https://github.com/apache/pulsar/pull/19470
> > > --
> > >
> > > 
> > >
> > > Pengcheng Jiang
> > >
> > > Software Engineer
> > >
> > > e: pengcheng.ji...@streamnative.io
> > >
> > > p: 13540631948
> > >
> > > streamnative.io
> > >
> > > 
> > > 
> > > 
> > >
>


-- 



Pengcheng Jiang

Software Engineer

e: pengcheng.ji...@streamnative.io

p: 13540631948

streamnative.io

Re: [DISCUSS] ClientVersion conflict across multiple language client

[Zike Yang]

Hi, Michael,

> I don't think we need to make it configurable because the library
owner, not the user, has full control when creating the Connect
command.

The Nodejs, Python, and C client are all based on the C++ client. Same
for the Scala client which is based on the Java client. They have no
control when creating the CommandConnect. But we need to make them
able to set the client version to resolve this conflict issue. I think
it is still worth making it configurable.

WDYT? Do you have any ideas?

Thanks,
Zike Yang

On Fri, Feb 24, 2023 at 3:24 PM Zike Yang  wrote:
>
> > I think it makes sense to make the format
> "pulsar--" for official clients. We could then
> recommend third party clients replace "pulsar" with a relevant org
> name. There isn't a way to enforce the version string though, so these
> would only be recommendations.
>
> +1 for this format.
>
> > F#:
> I'm not exactly sure how to interpret this input.
> https://github.com/fsprojects/pulsar-client-dotnet/blob/5c6dcff0184f87005f55dcc847893a889aaee2ad/src/Pulsar.Client/Internal/ClientCnx.fs#L119
>
> The format is like `2.11.0.0`. It's consistent with the current java
> client version format.
>
> > Because we filter out the official go client, I propose that we remove
> the filtering of certain client versions:
> https://github.com/apache/pulsar/pull/19616.
>
> Thanks for your PR. I have approved.
>
> Since there are no objections to this discussion, I will next make the
> client version format of all clients(at least official clients)
> consistent.
>
> BR,
> Zike Yang
>
> On Fri, Feb 24, 2023 at 3:22 AM Michael Marshall  wrote:
> >
> > > Go:
> > > ClientVersionString = "Pulsar Go " + Version
> > > https://github.com/apache/pulsar-client-go/blob/dedbdc45c63b06e6a12356785418cd906d6bab3c/pulsar/internal/version.go#L43
> >
> > Because we filter out the official go client, I propose that we remove
> > the filtering of certain client versions:
> > https://github.com/apache/pulsar/pull/19616.
> >
> > Thanks,
> > Michael
> >
> > On Thu, Feb 23, 2023 at 12:45 PM Michael Marshall  
> > wrote:
> > >
> > > I found another data point in favor of moving in this direction. Our
> > > protocol spec says the following about the client_version field:
> > >
> > > message CommandConnect {
> > > "client_version" : "Pulsar-Client-Java-v1.15.2",
> > > "auth_method_name" : "my-authentication-plugin",
> > > "auth_data" : "my-auth-data",
> > > "protocol_version" : 6
> > > }
> > >
> > > * `client_version`: String-based identifier. Format is not enforced.
> > >
> > > https://github.com/apache/pulsar-site/blame/main/docs/developing-binary-protocol.md#L133
> > >
> > > I think it makes sense to make the format
> > > "pulsar--" for official clients. We could then
> > > recommend third party clients replace "pulsar" with a relevant org
> > > name. There isn't a way to enforce the version string though, so these
> > > would only be recommendations.
> > >
> > > For official clients, here are our version strings:
> > >
> > > Go:
> > > ClientVersionString = "Pulsar Go " + Version
> > > https://github.com/apache/pulsar-client-go/blob/dedbdc45c63b06e6a12356785418cd906d6bab3c/pulsar/internal/version.go#L43
> > >
> > > C++
> > > connect->set_client_version(PULSAR_VERSION_STR);
> > > https://github.com/apache/pulsar-client-cpp/blob/96507cecdc90f10eca5febc48a0623f72d338615/lib/Commands.cc#L269
> > > std::string version = std::string("Pulsar-CPP-v") + PULSAR_VERSION_STR;
> > > https://github.com/apache/pulsar-client-cpp/blob/05807bdaf3a4341b22efd7c71f7b00c47cc31413/lib/HTTPLookupService.cc#L195
> > >
> > > I took a quick survey of some existing third party clients. Here are
> > > the results for how the version string is written:
> > >
> > > Rust:
> > > client_version: String::from("2.0.1-incubating"),
> > > https://github.com/streamnative/pulsar-rs/blob/eb87ef796bd8cb42fc72bb72ed14751fffa437e1/src/connection.rs#L1173
> > >
> > > Haskell:
> > > & F.clientVersion .~ "Pulsar-Client-Haskell-v" <> T.pack (showVersion 
> > > version)
> > > https://github.com/cr-org/supernova/blob/602409a18f47a38541ba24f5e885199efd383f48/lib/src/Pulsar/Protocol/Commands.hs#L22
> > >
> > > PHP:
> > > $cmd->setClientVersion(sprintf('ikilobyte/pulsar-client-php@v%s',
> > > Client::VERSION_ID));
> > > https://github.com/ikilobyte/pulsar-client-php/blob/e8847c62d3bf136886312a14a04f51c59ca99886/src/IO/StreamIO.php#L77
> > >
> > > F#:
> > > I'm not exactly sure how to interpret this input.
> > > https://github.com/fsprojects/pulsar-client-dotnet/blob/5c6dcff0184f87005f55dcc847893a889aaee2ad/src/Pulsar.Client/Internal/ClientCnx.fs#L119
> > >
> > > Scala:
> > > All appear to wrap the Pulsar Java client (or don't have a
> > > clientVersion string in their code base).
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Thu, Feb 23, 2023 at 11:39 AM Michael Marshall  
> > > wrote:
> > > >
> > > > > A better solution is that we could allow the
> > > > > user to specify the suffix of the client version.
> > > >
> > > > I 

Re: [DISCUSS] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

[Rui Fu]

+1

Best,

Rui Fu
On Feb 28, 2023 at 05:35 +0800, laminar , wrote:
> Hi all,
>
> I would like to discuss supporting the processingGuarantess 
> `EFFECTIVELY_ONCE` in python function runtime.
>
> In this PR's(https://github.com/apache/pulsar/pull/18929) discussion, we 
> conclude that to achieve the exactly processing guarantees for 
> `EFFECTIVELY_ONCE`, the user needs to ensure that the following 
> pre-requisites are met.
>
> 1. deduplication is enabled
> 2. set ProcessingGuarantees to EFFECTIVELY_ONCE
> 3. the function has only one source topic and one sink topic (both are 
> non-partitioned)
> 4. if partitioned topic is enabled, ensure that the number of partitions (of 
> both source and sink topics) is the same
>
> Currently, neither the python function runtime nor the java function runtime 
> can support the `EFFECTIVELY_ONCE` processing guarantee when using 
> partitioned topics.
>
> So in order to make python functions support `EFFECTIVELY_ONCE` processing 
> guarantee, I think we can introduce this feature incrementally, i.e. support 
> the `EFFECTIVELY_ONCE` processing guarantee for non-partitioned topics first.
>
> Then follow up with Rui’s 
> suggestion(https://github.com/apache/pulsar/pull/18929#issuecomment-1445977320)
>  to improve this feature.
>
>

Re: [DISCUSS] Support custom compressionType for pulsar functions

[Rui Fu]

+1, very useful, just one question: why not set the `LZ4` as the “zero” value 
instead? Like for the enum with following orders: `LZ4`, `NONE`, `ZLIB`, `ZSTD` 
and `SNAPPY`? So it will remain the backward compatibility.

Best,

Rui Fu
On Mar 1, 2023 at 10:18 +0800, Neng Lu , wrote:
> +1 for the change.
>
> On 2023/02/28 01:06:51 Pengcheng Jiang wrote:
> > Hello, community:
> >
> > ### Motivation
> >
> > Currently, pulsar functions are using `LZ4` as the compression type, and
> > users cannot change it, yet some users may want to custom this behavior.
> >
> > ### Modifications
> >
> > Add a `CompressionType` field(which is an enum) to the `ProducerSpec` in
> > the `Function.proto`; this enum has six values: `NOTSET`, `NONE`, `LZ4`,
> > `ZLIB`, `ZSTD` and `SNAPPY`, there is a `NOTSET` value besides of 5
> > supported compression type, so that even users don't set the
> > `CompressionType`, it will fallback to its "zero" value: `NOTSET` instead
> > of `NONE`, and in such case, pulsar function instances will use `LZ4` to
> > keep the same behavior with before.
> >
> > PTAL when you have time and feel free to leave any comments.
> >
> > Best Regards,
> > Pengcheng Jiang
> >
> > [0] https://github.com/apache/pulsar/pull/19470
> > --
> >
> > 
> >
> > Pengcheng Jiang
> >
> > Software Engineer
> >
> > e: pengcheng.ji...@streamnative.io
> >
> > p: 13540631948
> >
> > streamnative.io
> >
> > 
> > 
> > 
> >

Re: [DISCUSS] Support custom compressionType for pulsar functions

[Neng Lu]

+1 for the change.

On 2023/02/28 01:06:51 Pengcheng Jiang wrote:
> Hello, community:
> 
> ### Motivation
> 
> Currently, pulsar functions are using `LZ4` as the compression type, and
> users cannot change it, yet some users may want to custom this behavior.
> 
> ### Modifications
> 
> Add a `CompressionType` field(which is an enum) to the `ProducerSpec` in
> the `Function.proto`; this enum has six values: `NOTSET`, `NONE`, `LZ4`,
> `ZLIB`, `ZSTD` and `SNAPPY`, there is a `NOTSET` value besides of 5
> supported compression type, so that even users don't set the
> `CompressionType`, it will fallback to its "zero" value: `NOTSET` instead
> of `NONE`, and in such case, pulsar function instances will use `LZ4` to
> keep the same behavior with before.
> 
> PTAL when you have time and feel free to leave any comments.
> 
> Best Regards,
> Pengcheng Jiang
> 
> [0] https://github.com/apache/pulsar/pull/19470
> -- 
> 
> 
> 
> Pengcheng Jiang
> 
> Software Engineer
> 
> e: pengcheng.ji...@streamnative.io
> 
> p: 13540631948
> 
> streamnative.io
> 
> 
> 
> 
> 

[Discussion] Allowing configure if function consumer should skip to latest

[Neng Lu]

Hello Community,

In this [PR](https://github.com/apache/pulsar/pull/17214), we changed the
function protobuf by adding one more field `bool skipToLatest = 14;`.

The change itself is minimum and self-contained for function internal usage.

Given the new community guideline that protobuf change should notify the
community, I'm writing this email to initiate the (late) discussion.

### Motivation
In certain failure cases, the function needs to skip all the content
between the last successfully acked message and the latest message in the
topic in order to skip the huge backlog and quick recovery.

### Modifications
1. Providing a boolean config for function submission cmd
2. PulsarSource will call `consumer.seek(MessageId.latest)` if the skip
flag is set
3. the internal function protobuf file will have a new field `skipToLatest`

Let me know your thoughts. And if there're big concerns regarding this
change, we will revert the above PR fist and then make modifications to it.

Best Regards,
Neng Lu

Re: [DISCUSS] PIP-249: Pulsar website redesign

[tison]

Hi Asaf,

Here are my two coins:

1. Big +1 on the proposed change to a neutral background. The blue
background makes color coordination quite difficult.
2. Since we already stick the top menu on scrolling, perhaps we don't have
to change the footer to duplicate nav items. At least the proposed look is
quite obsolete to have three blocks in the footer.

For community links, we have already a community button in the top nav bar,
and for the ASF footer display, https://kafka.apache.org/ provides a
solution to minimize it.

Best,
tison.


Asaf Mesika  于2023年3月1日周三 05:11写道：

> So far we have 4 people giving feedback: Yu, Dave, Enrico and Kyril.
>
> Anybody else wishes to express their feedback about the content of the PIP?
>
> On Mon, Feb 27, 2023 at 3:08 PM Enrico Olivelli 
> wrote:
>
> > I am supportive of this idea about the website.
> > I won't enter the details, but I have read the PIP and overall the
> > idea seems good to me
> >
> > Enrico
> >
> > Il giorno ven 24 feb 2023 alle ore 17:37 Asaf Mesika
> >  ha scritto:
> > >
> > > You are correct - we did start with what content we wanted to show,
> then
> > > designed it to fit that content.
> > > We are releasing it for discussion, iteration and approval, piece by
> > piece,
> > > as I said before since it's easier for everyone reviewing it do it in
> > small
> > > pieces (just like in code).
> > >
> > > The “Homepage structure” point is defined in the PIP’s scope, so I
> > thought
> > > > it meant content changing is implied at this step.
> > > >
> > >
> > > The "homepage structure" is detailed in the PIP and composed of:
> > >
> > >- News banner
> > >- Testimonials
> > >- Menu
> > >- Footer
> > >
> > > As you see it does not mean content changing as this, as we leave that
> to
> > > future steps.
> > >
> > >
> > >
> > > On Fri, Feb 24, 2023 at 4:51 PM Kiryl Valkovich
> > 
> > > wrote:
> > >
> > > > Asaf, okay. I see you have some plan.
> > > >
> > > > I’m just a bit confused with the steps ordering.
> > > > Usually, such work starts with “What content do we want to show?”
> That
> > > > implies content changes.
> > > > Then “How can we display this content in the best way?”. That implies
> > the
> > > > work on visual design and content adjustment.
> > > > Of course, this order is not mandatory, but it usually allows to save
> > time
> > > > on the number of iterations.
> > > > The “Homepage structure” point is defined in the PIP’s scope, so I
> > thought
> > > > it meant content changing is implied at this step.
> > > > Waiting for the finished site. :)
> > > > By the way, does someone have any more opinions on the topic?
> > > >
> > > > From: Asaf Mesika 
> > > > Date: Friday, February 24, 2023 at 3:06 PM
> > > > To: dev@pulsar.apache.org 
> > > > Subject: Re: [DISCUSS] PIP-249: Pulsar website redesign
> > > > I totally agree with your point Kyril, and as I wrote in the welcome
> > email
> > > > - I want to do all you wrote, but it has to be in bite size pieces.
> > This
> > > > piece is about theme and general look and feel. We "force" ourselves,
> > by
> > > > design, to avoid any content change - just like in code in this
> > proposal -
> > > > so people can focus only on the theme change. It's like in code - you
> > take
> > > > an epic, break it down to stories, and each is broken down to small
> > PRs.
> > > >
> > > > So yes, I have a pipeline of content changes:
> > > > * Landing page:
> > > > ** "Tell me about the pulsar community" section - containing exactly
> > what
> > > > you wanted. We gather great statistics that show the growing
> community
> > in
> > > > nice numbers, which can also feature company names
> > > > ** "Use cases and applications" - so people can know how Pulsar is
> > used by
> > > > companies (small and big)
> > > > and more.
> > > >
> > > > * Documentation:
> > > >As you probably know - the documentation itself is an
> encyclopedia,
> > so I
> > > > planned next week a PIP presenting a new structure (super high level)
> > and
> > > > presenting in detail the first section I'd like to focus on:
> > Quick-start
> > > > Guide.
> > > >
> > > > Each such change will be open for discussion to make it the best
> > possible
> > > > addition to the site.
> > > >
> > > > I'm happy you liked the look and feel (design) proposed (Emidio is
> > happy
> > > > too).
> > > >
> > > >
> > > > On Fri, Feb 24, 2023 at 10:21 AM Kiryl Valkovich
> > > > 
> > > > wrote:
> > > >
> > > > > While I find the proposed visual changes okay, I'm unsure about the
> > > > > homepage structure.
> > > > >
> > > > > I'd add one obvious point to the Goal section that can be
> formulated
> > > > > differently, but eventually is:
> > > > > - **The goal is to get more companies to start using Pulsar.**
> > > > >
> > > > > In other words, the landing page should **sell** the project to
> > potential
> > > > > users.
> > > > >
> > > > > The defined scope of this PIP includes the "Homepage structure"
> > point. I
> > > > > will try to 

Re: [DISCUSS] PIP-249: Pulsar website redesign

[Asaf Mesika]

So far we have 4 people giving feedback: Yu, Dave, Enrico and Kyril.

Anybody else wishes to express their feedback about the content of the PIP?

On Mon, Feb 27, 2023 at 3:08 PM Enrico Olivelli  wrote:

> I am supportive of this idea about the website.
> I won't enter the details, but I have read the PIP and overall the
> idea seems good to me
>
> Enrico
>
> Il giorno ven 24 feb 2023 alle ore 17:37 Asaf Mesika
>  ha scritto:
> >
> > You are correct - we did start with what content we wanted to show, then
> > designed it to fit that content.
> > We are releasing it for discussion, iteration and approval, piece by
> piece,
> > as I said before since it's easier for everyone reviewing it do it in
> small
> > pieces (just like in code).
> >
> > The “Homepage structure” point is defined in the PIP’s scope, so I
> thought
> > > it meant content changing is implied at this step.
> > >
> >
> > The "homepage structure" is detailed in the PIP and composed of:
> >
> >- News banner
> >- Testimonials
> >- Menu
> >- Footer
> >
> > As you see it does not mean content changing as this, as we leave that to
> > future steps.
> >
> >
> >
> > On Fri, Feb 24, 2023 at 4:51 PM Kiryl Valkovich
> 
> > wrote:
> >
> > > Asaf, okay. I see you have some plan.
> > >
> > > I’m just a bit confused with the steps ordering.
> > > Usually, such work starts with “What content do we want to show?” That
> > > implies content changes.
> > > Then “How can we display this content in the best way?”. That implies
> the
> > > work on visual design and content adjustment.
> > > Of course, this order is not mandatory, but it usually allows to save
> time
> > > on the number of iterations.
> > > The “Homepage structure” point is defined in the PIP’s scope, so I
> thought
> > > it meant content changing is implied at this step.
> > > Waiting for the finished site. :)
> > > By the way, does someone have any more opinions on the topic?
> > >
> > > From: Asaf Mesika 
> > > Date: Friday, February 24, 2023 at 3:06 PM
> > > To: dev@pulsar.apache.org 
> > > Subject: Re: [DISCUSS] PIP-249: Pulsar website redesign
> > > I totally agree with your point Kyril, and as I wrote in the welcome
> email
> > > - I want to do all you wrote, but it has to be in bite size pieces.
> This
> > > piece is about theme and general look and feel. We "force" ourselves,
> by
> > > design, to avoid any content change - just like in code in this
> proposal -
> > > so people can focus only on the theme change. It's like in code - you
> take
> > > an epic, break it down to stories, and each is broken down to small
> PRs.
> > >
> > > So yes, I have a pipeline of content changes:
> > > * Landing page:
> > > ** "Tell me about the pulsar community" section - containing exactly
> what
> > > you wanted. We gather great statistics that show the growing community
> in
> > > nice numbers, which can also feature company names
> > > ** "Use cases and applications" - so people can know how Pulsar is
> used by
> > > companies (small and big)
> > > and more.
> > >
> > > * Documentation:
> > >As you probably know - the documentation itself is an encyclopedia,
> so I
> > > planned next week a PIP presenting a new structure (super high level)
> and
> > > presenting in detail the first section I'd like to focus on:
> Quick-start
> > > Guide.
> > >
> > > Each such change will be open for discussion to make it the best
> possible
> > > addition to the site.
> > >
> > > I'm happy you liked the look and feel (design) proposed (Emidio is
> happy
> > > too).
> > >
> > >
> > > On Fri, Feb 24, 2023 at 10:21 AM Kiryl Valkovich
> > > 
> > > wrote:
> > >
> > > > While I find the proposed visual changes okay, I'm unsure about the
> > > > homepage structure.
> > > >
> > > > I'd add one obvious point to the Goal section that can be formulated
> > > > differently, but eventually is:
> > > > - **The goal is to get more companies to start using Pulsar.**
> > > >
> > > > In other words, the landing page should **sell** the project to
> potential
> > > > users.
> > > >
> > > > The defined scope of this PIP includes the "Homepage structure"
> point. I
> > > > will try to express my thoughts on this.
> > > >
> > > > ## Add "Trusted by" or "Used in production" section
> > > >
> > > > One of the best hooks to attract new customers is to show that their
> > > peers
> > > > or more prominent players use it.
> > > >
> > > > This is why I think it's essential to show the list of companies
> that use
> > > > Pulsar as closer to the page top, as possible.
> > > >
> > > > *Kafka in some form has this info on the first screen*
> > > >  > > >
> > >
> https://user-images.githubusercontent.com/9302460/221116492-b77cb8e9-eca1-46bd-b457-6889f4e708b6.png
> > > > >
> > > >
> > > > *Confluent has it on the second screen*
> > > >  > > >
> > >
> https://user-images.githubusercontent.com/9302460/221114783-4f22394e-d1fd-4640-8264-3b137f44e2a1.png
> > > > >
> > > >
> > > > *StreamNative has it on the second screen*
> > > >  > 

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Jiuming Tao]

Bump

> 2023年2月13日 14:56，Jiuming Tao  > 写道：
> 
> Hi all,
> 
> I would like to start a VOTE on `PIP-245: Make subscriptions of 
> non-persistent topic non-durable`.
> 
> Motivation:
> 
> There are two types of subscriptions for a topic: Durable and Non-durable.
> 
> We create a Consumer with a Durable subscription and a Reader with a 
> Non-durable subscription.
> 
> But for NonPersistentTopic, creating a Durable subscription is meaningless, 
> NonPersistentSubscription doesn't have a ManagedCursor to persistent its 
> data. After its consumer disconnected, the subscription couldn't be removed 
> automatically if we didn't set the value of subscriptionExpirationTimeMinutes 
> greater than 0.
> 
> For subscriptionExpirationTimeMinutes, it controls the subscription 
> expiration of NonPersistentTopic and PersistentTopic, if we set the value of 
> subscriptionExpirationTimeMinutes greater than 0, it may lead to data 
> loss(The durable subscriptions of PersistentTopic also can be removed).
> 
> And the Non-durable subscriptions will be removed automatically after all the 
> consumers disconnected, it's the existing logic.
> 
> For the purpose of removing the subscriptions which have no active consumers 
> of NonPersistentTopic and the above reasons, we can make all the 
> subscriptions of a NonPersistentTopic Non-durable.
> 
> 
> 
> For more details, you can read: https://github.com/apache/pulsar/issues/19448 
> 
> 
> And the discuss thread is available at: 
> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst 
> 
> 
> Thanks,
> Tao Jiuming

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Tao Jiuming]

bump

On 2023/02/13 06:56:09 Jiuming Tao wrote:
> Hi all,
> 
> I would like to start a VOTE on `PIP-245: Make subscriptions of 
> non-persistent topic non-durable`.
> 
> Motivation:
> 
> There are two types of subscriptions for a topic: Durable and Non-durable.
> 
> We create a Consumer with a Durable subscription and a Reader with a 
> Non-durable subscription.
> 
> But for NonPersistentTopic, creating a Durable subscription is meaningless, 
> NonPersistentSubscription doesn't have a ManagedCursor to persistent its 
> data. After its consumer disconnected, the subscription couldn't be removed 
> automatically if we didn't set the value of subscriptionExpirationTimeMinutes 
> greater than 0.
> 
> For subscriptionExpirationTimeMinutes, it controls the subscription 
> expiration of NonPersistentTopic and PersistentTopic, if we set the value of 
> subscriptionExpirationTimeMinutes greater than 0, it may lead to data 
> loss(The durable subscriptions of PersistentTopic also can be removed).
> 
> And the Non-durable subscriptions will be removed automatically after all the 
> consumers disconnected, it's the existing logic.
> 
> For the purpose of removing the subscriptions which have no active consumers 
> of NonPersistentTopic and the above reasons, we can make all the 
> subscriptions of a NonPersistentTopic Non-durable.
> 
> 
> 
> For more details, you can read: https://github.com/apache/pulsar/issues/19448 
> 
> 
> And the discuss thread is available at: 
> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst 
> 
> 
> Thanks,
> Tao Jiuming

[GitHub] [pulsar-adapters] cbornet commented on pull request #40: Update to Pulsar 2.11.0

[via GitHub]

cbornet commented on PR #40:
URL: https://github.com/apache/pulsar-adapters/pull/40#issuecomment-1448551957

   It seems that if we update Scala, it works for Spark but not for the kafka 
0.8 client wrapper.
   The Kafka 0.8 client wrapper is very old, depends on a very old scala 
version (2.9), only works as replacement for 0.8.1.1 and for instance not for 
0.8.2.0.
   I don't have context for this lib so I'm not sure what to do. Should we drop 
support for it ? Or update to 0.8.2.0 (which we can make work with scala 2.11. 
Better, but still not 2.12 so not sure it works with JDK 17) ? Or something 
else ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] The GET_TOPICS_OF_NAMESPACE command should filter out system topics

[Baodi Shi]

Have another problem:

If the transaction is enabled, the transaction-related system topic will
return by the client(__transaction_buffer_snapshot,
__transaction_buffer_snapshot_segments,
__transaction_buffer_snapshot_indexes).

I see this PR[1] filters some transaction-related topics, but these are
missing. Is there a reason? /cc @丛搏 


   - https://github.com/apache/pulsar/pull/16533/files#r1120225388




Thanks,
Baodi Shi


在 2023年2月28日 22:37:42 上，Baodi Shi  写道：

> Yes, we can wait for the v2.11.1 release.
>
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年2月28日 22:25:39 上，Yunze Xu  写道：
>
>> I agree with Enrico that we should revert this change in 2.11. BTW, I
>> see https://github.com/apache/pulsar-client-go/pull/969 tried to
>> upgrade the Pulsar version to 2.11 for Go client. I think we'd better
>> skip this version. There is no need to upgrade the server version
>> unless we have to add a new feature that depends on the new version of
>> Pulsar.
>>
>> Thanks,
>> Yunze
>>
>> On Tue, Feb 28, 2023 at 9:38 PM Enrico Olivelli 
>> wrote:
>>
>>
>> Baodi,
>>
>>
>> Il giorno mar 28 feb 2023 alle ore 05:58 Baodi Shi 
>>
>> ha scritto:
>>
>> >
>>
>> > Hi,
>>
>> >
>>
>> > In v2.11.0, the system topic enables by default [1]. This causes the
>>
>> > `GET_TOPICS_OF_NAMESPACE ` command to redundantly return the system
>> topic.
>>
>> >
>>
>> > This causes some incompatibility issues: If consumers use pattern
>> subscribe
>>
>> > and the pattern is `/tenant/namespace/.*`, it will subscribe
>>
>> > `__change_events` topic.
>>
>> >
>>
>> > I suggest that GET_TOPICS_OF_NAMESPACE should filter out system
>> topics(or
>>
>> > add include-system-topic param,  like the HTTP interface[2]) to maintain
>>
>> > consistent behavior with versions before 2.11.0
>>
>>
>> I agree that the behaviour must be consistent with previous versions.
>>
>>
>> This is a fix that we should do on 2.11.1, otherwise users upgrading
>>
>> from 2.10 will have a
>>
>> bad surprise.
>>
>>
>> Enrico
>>
>>
>> >
>>
>> >
>>
>> > Refer:
>>
>> > - [1] https://github.com/apache/pulsar/pull/15619
>>
>> > - [2] https://github.com/apache/pulsar/pull/15410
>>
>> >
>>
>> >
>>
>> > Thanks,
>>
>> > Baodi Shi
>>
>>

Re: [DISCUSS] The GET_TOPICS_OF_NAMESPACE command should filter out system topics

[Enrico Olivelli]

Il giorno mar 28 feb 2023 alle ore 15:37 Baodi Shi 
ha scritto:
>
> Yes, we can wait for the v2.11.1 release.
>
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年2月28日 22:25:39 上，Yunze Xu  写道：
>
> > I agree with Enrico that we should revert this change in 2.11.

I am not sure that we should "revert" it.
Baodi's approach works for me (to filter those topics in
GET_TOPICS_OF_NAMESPACE)

it is very important that old clients behave the same both with old
Pulsar servers and with Pulsar 2.11

Enrico


BTW, I
> > see https://github.com/apache/pulsar-client-go/pull/969 tried to
> > upgrade the Pulsar version to 2.11 for Go client. I think we'd better
> > skip this version. There is no need to upgrade the server version
> > unless we have to add a new feature that depends on the new version of
> > Pulsar.
> >
> > Thanks,
> > Yunze
> >
> > On Tue, Feb 28, 2023 at 9:38 PM Enrico Olivelli 
> > wrote:
> >
> >
> > Baodi,
> >
> >
> > Il giorno mar 28 feb 2023 alle ore 05:58 Baodi Shi 
> >
> > ha scritto:
> >
> > >
> >
> > > Hi,
> >
> > >
> >
> > > In v2.11.0, the system topic enables by default [1]. This causes the
> >
> > > `GET_TOPICS_OF_NAMESPACE ` command to redundantly return the system
> > topic.
> >
> > >
> >
> > > This causes some incompatibility issues: If consumers use pattern
> > subscribe
> >
> > > and the pattern is `/tenant/namespace/.*`, it will subscribe
> >
> > > `__change_events` topic.
> >
> > >
> >
> > > I suggest that GET_TOPICS_OF_NAMESPACE should filter out system topics(or
> >
> > > add include-system-topic param,  like the HTTP interface[2]) to maintain
> >
> > > consistent behavior with versions before 2.11.0
> >
> >
> > I agree that the behaviour must be consistent with previous versions.
> >
> >
> > This is a fix that we should do on 2.11.1, otherwise users upgrading
> >
> > from 2.10 will have a
> >
> > bad surprise.
> >
> >
> > Enrico
> >
> >
> > >
> >
> > >
> >
> > > Refer:
> >
> > > - [1] https://github.com/apache/pulsar/pull/15619
> >
> > > - [2] https://github.com/apache/pulsar/pull/15410
> >
> > >
> >
> > >
> >
> > > Thanks,
> >
> > > Baodi Shi
> >
> >

[GitHub] [pulsar-adapters] cbornet commented on pull request #40: Update to Pulsar 2.11.0

[via GitHub]

cbornet commented on PR #40:
URL: https://github.com/apache/pulsar-adapters/pull/40#issuecomment-1448310450

   That's weird. It compiles fine on my machine...
   The Spark version used is very old and there could be an incompatibility 
with JDK17
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] The GET_TOPICS_OF_NAMESPACE command should filter out system topics

[Baodi Shi]

Yes, we can wait for the v2.11.1 release.


Thanks,
Baodi Shi


在 2023年2月28日 22:25:39 上，Yunze Xu  写道：

> I agree with Enrico that we should revert this change in 2.11. BTW, I
> see https://github.com/apache/pulsar-client-go/pull/969 tried to
> upgrade the Pulsar version to 2.11 for Go client. I think we'd better
> skip this version. There is no need to upgrade the server version
> unless we have to add a new feature that depends on the new version of
> Pulsar.
>
> Thanks,
> Yunze
>
> On Tue, Feb 28, 2023 at 9:38 PM Enrico Olivelli 
> wrote:
>
>
> Baodi,
>
>
> Il giorno mar 28 feb 2023 alle ore 05:58 Baodi Shi 
>
> ha scritto:
>
> >
>
> > Hi,
>
> >
>
> > In v2.11.0, the system topic enables by default [1]. This causes the
>
> > `GET_TOPICS_OF_NAMESPACE ` command to redundantly return the system
> topic.
>
> >
>
> > This causes some incompatibility issues: If consumers use pattern
> subscribe
>
> > and the pattern is `/tenant/namespace/.*`, it will subscribe
>
> > `__change_events` topic.
>
> >
>
> > I suggest that GET_TOPICS_OF_NAMESPACE should filter out system topics(or
>
> > add include-system-topic param,  like the HTTP interface[2]) to maintain
>
> > consistent behavior with versions before 2.11.0
>
>
> I agree that the behaviour must be consistent with previous versions.
>
>
> This is a fix that we should do on 2.11.1, otherwise users upgrading
>
> from 2.10 will have a
>
> bad surprise.
>
>
> Enrico
>
>
> >
>
> >
>
> > Refer:
>
> > - [1] https://github.com/apache/pulsar/pull/15619
>
> > - [2] https://github.com/apache/pulsar/pull/15410
>
> >
>
> >
>
> > Thanks,
>
> > Baodi Shi
>
>

Re: [DISCUSS] The GET_TOPICS_OF_NAMESPACE command should filter out system topics

[Yunze Xu]

I agree with Enrico that we should revert this change in 2.11. BTW, I
see https://github.com/apache/pulsar-client-go/pull/969 tried to
upgrade the Pulsar version to 2.11 for Go client. I think we'd better
skip this version. There is no need to upgrade the server version
unless we have to add a new feature that depends on the new version of
Pulsar.

Thanks,
Yunze

On Tue, Feb 28, 2023 at 9:38 PM Enrico Olivelli  wrote:
>
> Baodi,
>
> Il giorno mar 28 feb 2023 alle ore 05:58 Baodi Shi 
> ha scritto:
> >
> > Hi,
> >
> > In v2.11.0, the system topic enables by default [1]. This causes the
> > `GET_TOPICS_OF_NAMESPACE ` command to redundantly return the system topic.
> >
> > This causes some incompatibility issues: If consumers use pattern subscribe
> > and the pattern is `/tenant/namespace/.*`, it will subscribe
> > `__change_events` topic.
> >
> > I suggest that GET_TOPICS_OF_NAMESPACE should filter out system topics(or
> > add include-system-topic param,  like the HTTP interface[2]) to maintain
> > consistent behavior with versions before 2.11.0
>
> I agree that the behaviour must be consistent with previous versions.
>
> This is a fix that we should do on 2.11.1, otherwise users upgrading
> from 2.10 will have a
> bad surprise.
>
> Enrico
>
> >
> >
> > Refer:
> > - [1] https://github.com/apache/pulsar/pull/15619
> > - [2] https://github.com/apache/pulsar/pull/15410
> >
> >
> > Thanks,
> > Baodi Shi

Re: [DISCUSS] The GET_TOPICS_OF_NAMESPACE command should filter out system topics

[Enrico Olivelli]

Baodi,

Il giorno mar 28 feb 2023 alle ore 05:58 Baodi Shi 
ha scritto:
>
> Hi,
>
> In v2.11.0, the system topic enables by default [1]. This causes the
> `GET_TOPICS_OF_NAMESPACE ` command to redundantly return the system topic.
>
> This causes some incompatibility issues: If consumers use pattern subscribe
> and the pattern is `/tenant/namespace/.*`, it will subscribe
> `__change_events` topic.
>
> I suggest that GET_TOPICS_OF_NAMESPACE should filter out system topics(or
> add include-system-topic param,  like the HTTP interface[2]) to maintain
> consistent behavior with versions before 2.11.0

I agree that the behaviour must be consistent with previous versions.

This is a fix that we should do on 2.11.1, otherwise users upgrading
from 2.10 will have a
bad surprise.

Enrico

>
>
> Refer:
> - [1] https://github.com/apache/pulsar/pull/15619
> - [2] https://github.com/apache/pulsar/pull/15410
>
>
> Thanks,
> Baodi Shi

Re: Introducer Pulsar admin api to pulsar-client-go

[Enrico Olivelli]

Il giorno mar 28 feb 2023 alle ore 10:40 ZhangJian He
 ha scritto:
>
> ping @PengHuiLi @Enrico Olivelli
>
> Thanks
> ZhangJian He
>
>
> On Mon, 27 Feb 2023 at 14:02, ZhangJian He  wrote:
>>
>> > Another idea is whether it is possible to generate clients for various
>> languages based on the current pulsar rest swagger files
>> Yes. But I don't think this is a suitable choice for the go client. Due to 
>> the existing swagger file not fully complying with the specification, it 
>> cannot be used directly to generate client code. Personally, I don't think 
>> fixing it is particularly easy and there is currently no one working on it.
>> For the go client, I would still recommend creating a new repository and 
>> having everyone contribute to it from there. WDYT?

I agree with this approach

Enrico

>>
>> Also, I would appreciate it if others could let me know your thoughts on 
>> this matter. @PengHuiLi @Enrico Olivelli
>>
>> Thanks
>> ZhangJian He
>>
>>
>> On Sun, 26 Feb 2023 at 21:04, Guangning E  wrote:
>>>
>>> Another idea is whether it is possible to generate clients for various
>>> languages based on the current pulsar rest swagger files?
>>>
>>> Thanks,
>>> Guangning
>>>
>>> On Sat, Feb 25, 2023 at 1:44 PM Zixuan Liu  wrote:
>>>
>>> > > I would like to suggest creating a new repository called
>>> > "pulsar-admin-go"
>>> >
>>> > +1, I agreed with you.
>>> >
>>> > Thanks,
>>> > Zixuan
>>> >
>>> > ZhangJian He  于2023年2月25日周六 11:05写道：
>>> >
>>> > > Following our previous discussions via email, I would like to suggest
>>> > > creating a new repository called "pulsar-admin-go". This repository will
>>> > > serve as a platform for all of us to contribute our code. Pulsarctl[0] 
>>> > > go
>>> > > first.
>>> > >
>>> > > Please let me know your thoughts on this matter.
>>> > >
>>> > > [0] - https://github.com/streamnative/pulsarctl
>>> > >
>>> > > Thanks
>>> > > ZhangJian He
>>> > >
>>> > >
>>> > > On Fri, 17 Feb 2023 at 23:24, ZhangJian He  wrote:
>>> > >
>>> > > > Thank for StreamNative for willing to donate this project. This means
>>> > we
>>> > > > don't have to develop and maintain a set of HTTP code from scratch. My
>>> > > idea
>>> > > > aligns with Yunze's, and separating it into a standalone
>>> > pulsar-admin-go
>>> > > > project would be better. The **pulsarctl** repo contains bookkeeper
>>> > http
>>> > > > call too. Maybe we can have a project bookkeeper-admin-go ?（it's a
>>> > liitle
>>> > > > going off-topic ）
>>> > > >
>>> > > > Thanks
>>> > > > ZhangJian He
>>> > > >
>>> > > >
>>> > > > On Fri, 17 Feb 2023 at 20:29, PengHui Li 
>>> > > wrote:
>>> > > >
>>> > > >> Hi Yunze,
>>> > > >>
>>> > > >> Yes, we can split it.
>>> > > >> Both one repo with two modules or two repos works for me.
>>> > > >>
>>> > > >> The pulsarctl already have the admin API and CLI.
>>> > > >> So I think we don’t need to develop another one.
>>> > > >>
>>> > > >> Best,
>>> > > >> Penghui
>>> > > >>
>>> > > >> > On Feb 17, 2023, at 17:44, Yunze Xu 
>>> > > >> wrote:
>>> > > >> >
>>> > > >> > Hi PengHui,
>>> > > >> >
>>> > > >> > Now I changed my mind a bit. Even if the pulsarctl was contributed
>>> > to
>>> > > >> > the Apache Foundation, I think we should also avoid adding it as 
>>> > > >> > the
>>> > > >> > dependency. What we need is an API layer but not the CLI, while
>>> > > >> > pulsarctl couples the API and CLI.
>>> > > >> >
>>> > > >> > At the moment, my expectation is:
>>> > > >> > 1. Use a separate repo (e.g. pulsar-admin-go) to implement the 
>>> > > >> > admin
>>> > > >> > APIs in Golang.
>>> > > >> > 2. Depend this new repo in pulsarctl.
>>> > > >> >
>>> > > >> > Then we will have three Go projects:
>>> > > >> > - pulsar-client-go: The Pulsar Go client APIs
>>> > > >> > - pulsar-admin-go: The Pulsar Go admin APIs
>>> > > >> > - pulsarctl: The admin CLI tool written in Go
>>> > > >> >
>>> > > >> > Thanks,
>>> > > >> > Yunze
>>> > > >> >
>>> > > >> > On Fri, Feb 17, 2023 at 4:22 PM PengHui Li 
>>> > > wrote:
>>> > > >> >>
>>> > > >> >> I checked with Sijie today.
>>> > > >> >> StreamNative can contribute the pulsarctl project to Apache
>>> > > Foundation.
>>> > > >> >>
>>> > > >> >> Regards,
>>> > > >> >> Penghui
>>> > > >> >>
>>> > > >> >> On Fri, Feb 17, 2023 at 4:02 PM Enrico Olivelli <
>>> > eolive...@gmail.com
>>> > > >
>>> > > >> wrote:
>>> > > >> >>
>>> > > >> >>> I agree to add an admin API to the go client, this would be very
>>> > > >> helpful.
>>> > > >> >>>
>>> > > >> >>> Il giorno ven 17 feb 2023 alle ore 08:44 Zixuan Liu
>>> > > >> >>>  ha scritto:
>>> > > >> 
>>> > > >>  Hi Zhangjian,
>>> > > >> 
>>> > > >>  This is a good idea to write the admin client by golang, but I
>>> > > don't
>>> > > >>  suggest add the admin features to pulsar-go-client, it's better
>>> > to
>>> > > >> use a
>>> > > >>  new repository to do that to  separate dependencies.
>>> > > >> 
>>> > > >>  BTW, StreamNative has a pulsarctl [0] tool, which includes the
>>> > > admin
>>> > > >> api.

[GitHub] [pulsar-adapters] eolivelli commented on pull request #40: Update to Pulsar 2.11.0

[via GitHub]

eolivelli commented on PR #40:
URL: https://github.com/apache/pulsar-adapters/pull/40#issuecomment-1448087003

   LGTM as far as we fix the build issues :-) 
   
   ```
   Error:  Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) 
on project pulsar-spark: Compilation failure
   Error:  
/home/runner/work/pulsar-adapters/pulsar-adapters/pulsar-spark/src/main/java/org/apache/pulsar/spark/SparkStreamingPulsarReceiver.java:[41,7]
 error: cannot access Serializable
   Error:class file for scala.Serializable not found
   Error:  -> [Help 1]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-adapters] cbornet opened a new pull request, #40: Update to Pulsar 2.11.0

[via GitHub]

cbornet opened a new pull request, #40:
URL: https://github.com/apache/pulsar-adapters/pull/40

   ### Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
 - If a feature is not applicable for documentation, explain why?
 - If a feature is not documented yet in this PR, please create a followup 
issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Please stop cherry-picking (breaking) changes to the released branches

[Enrico Olivelli]

Il giorno mar 28 feb 2023 alle ore 12:11 PengHui Li
 ha scritto:
>
> > By the way the main point in this email thread is that we should
> totally stop to do cherry-picks of stuff that it is
> not strictly needed
>
> Yes, the main issue we need to resolve is how we can define if
>  the stuff strictly needed to cherry-pick. Do you think the author
> to provide the cherry-pick information or reviewers to add labels
> and confirm the label is correct before merging it is a good way?
> Who wants to update the release/* label, the context is required.
> Do not only change the release label without any information.

Maybe it is better to start a discussion on the mailing list when you want to
cherry-pick something and wait for some time.
If nobody objects then it is good to go

Like we slowed down the pace to add new big changes with PIP
we could follow a slower workflow for cherry-picks.

We could try this strategy for a while.

After all, we are never in a hurry to merge a patch (urgent patches,
for security issues follow a different path),
we aren't cutting releases very often.


Enrico

>
> Or, push PR for every cherry-pick to get approved by committers.
>
> Thanks,
> Penghui
>
> On Tue, Feb 28, 2023 at 6:32 PM Enrico Olivelli  wrote:
>
> > Il giorno mar 28 feb 2023 alle ore 11:19 Yubiao Feng
> >  ha scritto:
> > >
> > > Append asuggestion:
> > > - After a PR revert, we need to remove the label named "release-xxx",
> > which
> > > can alleviate the release manager's work
> >
> > I think that it is up to the committer who merges the patch to
> > cherry-pick immediately to the other branches.
> > At that point you have enough context to merge the patch and for sure
> > the committer knows the patch well.
> >
> > In Apache BookKeeper and in Apache ZooKeeper we have a script that
> > does the merge against the target branch and
> > then it allows you to cherry-pick the other branches.
> >
> > Delaying the merge too much makes things harder.
> >
> > By the way the main point in this email thread is that we should
> > totally stop to do cherry-picks of stuff that it is
> > not strictly needed
> >
> >
> > Enrico
> >
> > >
> > > Thanks
> > > Yubiao Feng
> > >
> > > On Mon, Feb 27, 2023 at 11:27 PM Enrico Olivelli 
> > > wrote:
> > >
> > > > Hello Committers,
> > > > I believe that we should stop cherry-picking breaking changes like [1]
> > > > to released branches.
> > > > Really, this is something that we cannot do.
> > > >
> > > > When you decide to cherry-pick a commit to a "stable branch",
> > > > currently branch-2.8, branch-2.9, branch-2.10 and branch-2.11 you
> > > > always have to think about these things:
> > > > - is it a breaking change ?
> > > > - is it really needed ?
> > > > - could it mine the stability of the branch ?
> > > >
> > > > The answer is usually that you can cherry-pick a change only if it
> > > > falls into these categories:
> > > > - there is a security hole to fix (in this case the PMC has to deal
> > > > with it, and usually this is done not publicly)
> > > > - there is a bad bug that cause data loss or other serious problems
> > > >
> > > > I have sent this message a few other times in the past.
> > > > We, the Pulsar community, are responsible for the stability of the
> > > > project and product that our users use in production.
> > > >
> > > > Even if you think that something that could "improve the performance"
> > > > or "do something better" is appealing you always have to keep in mind
> > > > that the risk of breaking something that is stable is too high in
> > > > respect to the gain in terms of performances or anything else.
> > > >
> > > > Improvements should go only to the master branch, and users will
> > > > benefit from them when we will cut a release.
> > > >
> > > > This is a free OSS project on which many users count on.
> > > >
> > > > If you are eager to see a performance improvement in your system, then
> > > > this is fine,
> > > > this is OSS and you can legally have a fork and cherry-pick the
> > > > patches and build it on your own.
> > > > This is the reason why OSS is cool.
> > > > But if you are able to cherry-pick a patch you are also able to
> > > > maintain your fork and fix any problems if the patch caused a
> > > > regression.
> > > >
> > > > Most of the consumers of OSS products rely on us because they don't
> > > > have enough engineering resources to maintain such a project by
> > > > themselves.
> > > >
> > > > They trust us and they won't scan a list of tens of commits in order
> > > > to double check if the upgrade will change the behaviour of their
> > > > applications.
> > > >
> > > > This is Pulsar momentum, let's do our best to fulfill the expectations
> > > > of the companies that are adopting our project.
> > > >
> > > > Enrico
> > > >
> > > > [1]
> > > >
> > https://github.com/apache/pulsar/pull/19640#pullrequestreview-1315805022
> > > >
> >

Re: Please stop cherry-picking (breaking) changes to the released branches

[PengHui Li]

> By the way the main point in this email thread is that we should
totally stop to do cherry-picks of stuff that it is
not strictly needed

Yes, the main issue we need to resolve is how we can define if
 the stuff strictly needed to cherry-pick. Do you think the author
to provide the cherry-pick information or reviewers to add labels
and confirm the label is correct before merging it is a good way?
Who wants to update the release/* label, the context is required.
Do not only change the release label without any information.

Or, push PR for every cherry-pick to get approved by committers.

Thanks,
Penghui

On Tue, Feb 28, 2023 at 6:32 PM Enrico Olivelli  wrote:

> Il giorno mar 28 feb 2023 alle ore 11:19 Yubiao Feng
>  ha scritto:
> >
> > Append asuggestion:
> > - After a PR revert, we need to remove the label named "release-xxx",
> which
> > can alleviate the release manager's work
>
> I think that it is up to the committer who merges the patch to
> cherry-pick immediately to the other branches.
> At that point you have enough context to merge the patch and for sure
> the committer knows the patch well.
>
> In Apache BookKeeper and in Apache ZooKeeper we have a script that
> does the merge against the target branch and
> then it allows you to cherry-pick the other branches.
>
> Delaying the merge too much makes things harder.
>
> By the way the main point in this email thread is that we should
> totally stop to do cherry-picks of stuff that it is
> not strictly needed
>
>
> Enrico
>
> >
> > Thanks
> > Yubiao Feng
> >
> > On Mon, Feb 27, 2023 at 11:27 PM Enrico Olivelli 
> > wrote:
> >
> > > Hello Committers,
> > > I believe that we should stop cherry-picking breaking changes like [1]
> > > to released branches.
> > > Really, this is something that we cannot do.
> > >
> > > When you decide to cherry-pick a commit to a "stable branch",
> > > currently branch-2.8, branch-2.9, branch-2.10 and branch-2.11 you
> > > always have to think about these things:
> > > - is it a breaking change ?
> > > - is it really needed ?
> > > - could it mine the stability of the branch ?
> > >
> > > The answer is usually that you can cherry-pick a change only if it
> > > falls into these categories:
> > > - there is a security hole to fix (in this case the PMC has to deal
> > > with it, and usually this is done not publicly)
> > > - there is a bad bug that cause data loss or other serious problems
> > >
> > > I have sent this message a few other times in the past.
> > > We, the Pulsar community, are responsible for the stability of the
> > > project and product that our users use in production.
> > >
> > > Even if you think that something that could "improve the performance"
> > > or "do something better" is appealing you always have to keep in mind
> > > that the risk of breaking something that is stable is too high in
> > > respect to the gain in terms of performances or anything else.
> > >
> > > Improvements should go only to the master branch, and users will
> > > benefit from them when we will cut a release.
> > >
> > > This is a free OSS project on which many users count on.
> > >
> > > If you are eager to see a performance improvement in your system, then
> > > this is fine,
> > > this is OSS and you can legally have a fork and cherry-pick the
> > > patches and build it on your own.
> > > This is the reason why OSS is cool.
> > > But if you are able to cherry-pick a patch you are also able to
> > > maintain your fork and fix any problems if the patch caused a
> > > regression.
> > >
> > > Most of the consumers of OSS products rely on us because they don't
> > > have enough engineering resources to maintain such a project by
> > > themselves.
> > >
> > > They trust us and they won't scan a list of tens of commits in order
> > > to double check if the upgrade will change the behaviour of their
> > > applications.
> > >
> > > This is Pulsar momentum, let's do our best to fulfill the expectations
> > > of the companies that are adopting our project.
> > >
> > > Enrico
> > >
> > > [1]
> > >
> https://github.com/apache/pulsar/pull/19640#pullrequestreview-1315805022
> > >
>

Re: Please stop cherry-picking (breaking) changes to the released branches

[Enrico Olivelli]

Il giorno mar 28 feb 2023 alle ore 11:19 Yubiao Feng
 ha scritto:
>
> Append asuggestion:
> - After a PR revert, we need to remove the label named "release-xxx", which
> can alleviate the release manager's work

I think that it is up to the committer who merges the patch to
cherry-pick immediately to the other branches.
At that point you have enough context to merge the patch and for sure
the committer knows the patch well.

In Apache BookKeeper and in Apache ZooKeeper we have a script that
does the merge against the target branch and
then it allows you to cherry-pick the other branches.

Delaying the merge too much makes things harder.

By the way the main point in this email thread is that we should
totally stop to do cherry-picks of stuff that it is
not strictly needed


Enrico

>
> Thanks
> Yubiao Feng
>
> On Mon, Feb 27, 2023 at 11:27 PM Enrico Olivelli 
> wrote:
>
> > Hello Committers,
> > I believe that we should stop cherry-picking breaking changes like [1]
> > to released branches.
> > Really, this is something that we cannot do.
> >
> > When you decide to cherry-pick a commit to a "stable branch",
> > currently branch-2.8, branch-2.9, branch-2.10 and branch-2.11 you
> > always have to think about these things:
> > - is it a breaking change ?
> > - is it really needed ?
> > - could it mine the stability of the branch ?
> >
> > The answer is usually that you can cherry-pick a change only if it
> > falls into these categories:
> > - there is a security hole to fix (in this case the PMC has to deal
> > with it, and usually this is done not publicly)
> > - there is a bad bug that cause data loss or other serious problems
> >
> > I have sent this message a few other times in the past.
> > We, the Pulsar community, are responsible for the stability of the
> > project and product that our users use in production.
> >
> > Even if you think that something that could "improve the performance"
> > or "do something better" is appealing you always have to keep in mind
> > that the risk of breaking something that is stable is too high in
> > respect to the gain in terms of performances or anything else.
> >
> > Improvements should go only to the master branch, and users will
> > benefit from them when we will cut a release.
> >
> > This is a free OSS project on which many users count on.
> >
> > If you are eager to see a performance improvement in your system, then
> > this is fine,
> > this is OSS and you can legally have a fork and cherry-pick the
> > patches and build it on your own.
> > This is the reason why OSS is cool.
> > But if you are able to cherry-pick a patch you are also able to
> > maintain your fork and fix any problems if the patch caused a
> > regression.
> >
> > Most of the consumers of OSS products rely on us because they don't
> > have enough engineering resources to maintain such a project by
> > themselves.
> >
> > They trust us and they won't scan a list of tens of commits in order
> > to double check if the upgrade will change the behaviour of their
> > applications.
> >
> > This is Pulsar momentum, let's do our best to fulfill the expectations
> > of the companies that are adopting our project.
> >
> > Enrico
> >
> > [1]
> > https://github.com/apache/pulsar/pull/19640#pullrequestreview-1315805022
> >

Re: Please stop cherry-picking (breaking) changes to the released branches

[Yubiao Feng]

Append asuggestion:
- After a PR revert, we need to remove the label named "release-xxx", which
can alleviate the release manager's work

Thanks
Yubiao Feng

On Mon, Feb 27, 2023 at 11:27 PM Enrico Olivelli 
wrote:

> Hello Committers,
> I believe that we should stop cherry-picking breaking changes like [1]
> to released branches.
> Really, this is something that we cannot do.
>
> When you decide to cherry-pick a commit to a "stable branch",
> currently branch-2.8, branch-2.9, branch-2.10 and branch-2.11 you
> always have to think about these things:
> - is it a breaking change ?
> - is it really needed ?
> - could it mine the stability of the branch ?
>
> The answer is usually that you can cherry-pick a change only if it
> falls into these categories:
> - there is a security hole to fix (in this case the PMC has to deal
> with it, and usually this is done not publicly)
> - there is a bad bug that cause data loss or other serious problems
>
> I have sent this message a few other times in the past.
> We, the Pulsar community, are responsible for the stability of the
> project and product that our users use in production.
>
> Even if you think that something that could "improve the performance"
> or "do something better" is appealing you always have to keep in mind
> that the risk of breaking something that is stable is too high in
> respect to the gain in terms of performances or anything else.
>
> Improvements should go only to the master branch, and users will
> benefit from them when we will cut a release.
>
> This is a free OSS project on which many users count on.
>
> If you are eager to see a performance improvement in your system, then
> this is fine,
> this is OSS and you can legally have a fork and cherry-pick the
> patches and build it on your own.
> This is the reason why OSS is cool.
> But if you are able to cherry-pick a patch you are also able to
> maintain your fork and fix any problems if the patch caused a
> regression.
>
> Most of the consumers of OSS products rely on us because they don't
> have enough engineering resources to maintain such a project by
> themselves.
>
> They trust us and they won't scan a list of tens of commits in order
> to double check if the upgrade will change the behaviour of their
> applications.
>
> This is Pulsar momentum, let's do our best to fulfill the expectations
> of the companies that are adopting our project.
>
> Enrico
>
> [1]
> https://github.com/apache/pulsar/pull/19640#pullrequestreview-1315805022
>

Re: Please stop cherry-picking (breaking) changes to the released branches

[Zike Yang]

> If the author haven't provided the context, any committer who want to add
 the release/* label should left a comment about why the PR should be
cherry-pick.

Totally agree. The committer needs to add at least one comment for the
reason if he wants to add the release labels.

I think we could add this to the committer guideline if we reach this consensus.

Thanks,
Zike Yang

On Tue, Feb 28, 2023 at 3:19 PM PengHui Li  wrote:
>
> I found another example https://github.com/apache/pulsar/pull/19425
> Which has cherry-picked and then reverted
>
> If the PR's author knows the PR should be cherry-pick to some branches,
> it's better to contains this part in the PR description and explain why the
> fix
> should be cherry-picked. The reviewer should also review whether it is
> worth cherry-picking and then updating the labels.
>
> If the author haven't provided the context, any committer who want to add
>  the release/* label should left a comment about why the PR should be
> cherry-pick.
>
> I think it will helped the release manager to understand should or
> shouldn't
> to cherry-pick the PRs.
>
> We can also update the PR template
>
> Thanks,
> Penghui
>
>
> On Tue, Feb 28, 2023 at 1:36 PM Dave Fisher  wrote:
>
> >
> >
> > Sent from my iPhone
> >
> > > On Feb 27, 2023, at 9:28 PM, tison  wrote:
> > >
> > > 
> > >>
> > >> Yes, but..
> > >
> > > For this case, I agree that the RM trust Jason who tagged the PR needs to
> > > be picked to 2.10. In this case Jason was supposed to do the check.
> > >
> > >> guidance for release managers to evaluate PR cherry-pick labels
> > carefully
> > >
> > > For the current workflow, the RM can trust the labels since only
> > committers
> > > can label a PR.
> >
> > The RM should still verify what they are trusting. People make mistakes.
> > >
> > > We may add some guidelines in both label strategy and release process on
> > > the Contribution Guide to talk about breaking changes. Generally:
> > >
> > > * Default config value.
> > > * Public API breaking.
> > > * Dependency upgrading (may or may not a breaking change)
> > > * Other user-visible behavior changes (in the issued PR, it changes
> > whether
> > > an offload data is deleted on topic deleted)
> > >
> > > There's no automation/tests to cover all the cases so we still rely on
> > > proper reviews and tags.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Dave Fisher  于2023年2月28日周二 13:21写道：
> > >
> > >>
> > >>
> > >> Sent from my iPhone
> > >>
> >  On Feb 27, 2023, at 9:08 PM, tison  wrote:
> > >>>
> > >>> 
> > 
> >  The release manager is unable to review all PRs before releasing it.
> > >>>
> > >>> At least the RM is responsible for PRs cherry-picked he/she made. As we
> > >>> take compatibility in a high priority, if it's unclear a fix (patch)
> > >>> without breaking changes, the RM can ask for confirmation.
> > >>
> > >> Is there guidance for release managers to evaluate PR cherry-pick labels
> > >> carefully (how to confirm)?
> > >>
> > >> Best,
> > >> Dave
> > >>>
> > >>> Best,
> > >>> tison.
> > >>>
> > >>>
> > >>> PengHui Li  于2023年2月28日周二 12:45写道：
> > >>>
> >  Hi enrico,
> > 
> >  +1 for your point.
> > 
> >  Do you know the details of the breaking change?
> >  I can't find any discussions under the mailing list about the breaking
> >  change.
> > 
> >  I have added the `release/important-notice ` label to the PR, and we
> > >> should
> >  also discuss first, better to have a proposal if we are making
> > breaking
> >  changes.
> > 
> >  IMO, the main issue here is that the release manager doesn't know the
> > >> PR is
> >  introducing breaking changes, rather than thinking that the
> > >> introduction of
> >  breaking changes is reasonable to the patch release. I noticed Jason
> > had
> >  added the release/* label, I think he also isn't aware of the breaking
> >  change.
> > 
> >  The release manager is unable to review all PRs before releasing it.
> >  And the PR title said
> > 
> >  "[Fix][Tiered Storage] Eagerly Delete Offloaded Segments On Topic
> >  Deletion".
> > 
> >  My impression, it also should be bug fix.
> > 
> >  Regards,
> >  Penghui
> > 
> > > On Tue, Feb 28, 2023 at 10:32 AM Xiangying Meng <
> > xiangy...@apache.org>
> > > wrote:
> > >
> > > Hi Enrico Olivelli,
> > >
> > > Totally agree, we should be careful when cherry-picking PRs. And we
> > >> can't
> > > trust our own judgment too much. For an uncertain PR, we must submit
> > a
> > >> PR
> > > and wait for everyone to review it together.
> > > For example, for the PR [1] mentioned above, the measure I took was
> > to
> >  push
> > > a PR to cherry-pick and move it to the next release version (2.10.5)
> > so
> > > that we have enough time to discuss and reach an agreement.
> > >
> > > Sincerely,
> > > Xiangying
> > > [1]
> > >
> > >>
> > 

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

[ZhangJian He]

ping @Zixuan Liu @Enrico Olivelli 
So, I am not sure if we can update to 3.x. Maybe we will need provider two
versions of **pulsar-admin-client**?

Thanks
ZhangJian He


On Mon, 27 Feb 2023 at 16:37, ZhangJian He  wrote:

> Sorry, what I want to talk is `pulsar-admin-client`, it uses jersey
> client, which depends on `jakarta.ws.rs-api`
>
> Thanks
> ZhangJian He
>
>
> On Mon, 27 Feb 2023 at 16:27, Enrico Olivelli  wrote:
>
>> Are we talking about the Server side or the Pulsar Admin Client (that
>> uses HTTP) ?
>> On the server side we don't care about "SpringBoot", we are not using it
>>
>> Can you please elaborate more ?
>> Enrico
>>
>>
>> Il giorno lun 27 feb 2023 alle ore 09:02 Zixuan Liu
>>  ha scritto:
>> >
>> > Only the `canal` module requires the spring, so don't worry.
>> >
>> > If I wrong, please let me know.
>> >
>> > Thanks,
>> > Zixuan
>> >
>> > ZhangJian He  于2023年2月27日周一 14:04写道：
>> >
>> > > Hi @Enrico Olivelli  @Zixuan Liu
>> > >
>> > > I'm concerned that this change may create some complex compatibility
>> issues
>> > > between our client and Springboot. Springboot 2.x uses the package
>> name
>> > > "javax" while Springboot 3.x uses "jakarta". I'm still not entirely
>> sure
>> > > about the extent of its impact.
>> > >
>> > > Thanks
>> > > ZhangJian He
>> > >
>> > >
>> > > On Mon, 27 Feb 2023 at 13:12, Zixuan Liu  wrote:
>> > >
>> > > > > IIUC the main pain point is for Custom Servlet developers who will
>> > > > have to migrate to the jakarta.xxx packages
>> > > > is this correct ?
>> > > >
>> > > > Sure!
>> > > >
>> > > > Our web service and some http client depend on the jersey. I
>> checked on
>> > > > jetty and our project. Looks like we can only use the Jakarta EE9.
>> > > >
>> > > > We neet to upgrade the jetty to the 11.0.x from the 9.x and jersey
>> to
>> > > > 3.0.x. from 2.x.
>> > > >
>> > > > jetty 11.0.x - JakartaEE 9 - Java 11, you can see this in the
>> > > > https://www.eclipse.org/jetty/
>> > > > jersey 3.0.x - JakartaEE 9 - Java 8+(It looks like it supports Java
>> 8,
>> > > not
>> > > > sure how to implementation), you can see
>> > > > https://eclipse-ee4j.github.io/jersey/ and
>> > > >
>> > > >
>> > >
>> https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest30x/modules-and-dependencies.html#d0e560
>> > > >
>> > > > Thanks,
>> > > > Zixuan
>> > > >
>> > > > Enrico Olivelli  于2023年2月26日周日 20:48写道：
>> > > >
>> > > > > +1 to upgrade.
>> > > > >
>> > > > > IIUC the main pain point is for Custom Servlet developers who will
>> > > > > have to migrate to the jakarta.xxx packages
>> > > > > is this correct ?
>> > > > >
>> > > > > Enrico
>> > > > >
>> > > > > Il giorno sab 25 feb 2023 alle ore 06:46 Zixuan Liu
>> > > > >  ha scritto:
>> > > > > >
>> > > > > > Sounds greate that migrating to jersey 3.x from jersey 2.x.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Zixuan
>> > > > > >
>> > > > > > ZhangJian He  于2023年2月25日周六 11:01写道：
>> > > > > >
>> > > > > > > Hello, community. I want to know Which jakartaEE version we
>> plan
>> > > for
>> > > > > pulsar
>> > > > > > > 3.0.0. Now we use jersey 2.x, which uses Jakarta EE8.
>> > > > > > >
>> > > > > > > Maybe we will need to update the jersey.
>> > > > > > > Eclipse Jersey (eclipse-ee4j.github.io)
>> > > > > > > 
>> > > > > > > 2.x is for Jakarta EE8. 3.0.x is for Jakarta EE9 and 3.1.x is
>> for
>> > > > > Jakarta
>> > > > > > > EE10.
>> > > > > > >
>> > > > > > > Thank you for your time and consideration.
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > > ZhangJian He
>> > > > > > >
>> > > > >
>> > > >
>> > >
>>
>

Re: Introducer Pulsar admin api to pulsar-client-go

[ZhangJian He]

ping @PengHuiLi @Enrico Olivelli 

Thanks
ZhangJian He


On Mon, 27 Feb 2023 at 14:02, ZhangJian He  wrote:

> > Another idea is whether it is possible to generate clients for various
> languages based on the current pulsar rest swagger files
> Yes. But I don't think this is a suitable choice for the go client. Due to
> the existing swagger file not fully complying with the specification, it
> cannot be used directly to generate client code. Personally, I don't think
> fixing it is particularly easy and there is currently no one working on it.
> For the go client, I would still recommend creating a new repository and
> having everyone contribute to it from there. WDYT?
>
> Also, I would appreciate it if others could let me know your thoughts on
> this matter. @PengHuiLi @Enrico Olivelli 
>
> Thanks
> ZhangJian He
>
>
> On Sun, 26 Feb 2023 at 21:04, Guangning E  wrote:
>
>> Another idea is whether it is possible to generate clients for various
>> languages based on the current pulsar rest swagger files?
>>
>> Thanks,
>> Guangning
>>
>> On Sat, Feb 25, 2023 at 1:44 PM Zixuan Liu  wrote:
>>
>> > > I would like to suggest creating a new repository called
>> > "pulsar-admin-go"
>> >
>> > +1, I agreed with you.
>> >
>> > Thanks,
>> > Zixuan
>> >
>> > ZhangJian He  于2023年2月25日周六 11:05写道：
>> >
>> > > Following our previous discussions via email, I would like to suggest
>> > > creating a new repository called "pulsar-admin-go". This repository
>> will
>> > > serve as a platform for all of us to contribute our code.
>> Pulsarctl[0] go
>> > > first.
>> > >
>> > > Please let me know your thoughts on this matter.
>> > >
>> > > [0] - https://github.com/streamnative/pulsarctl
>> > >
>> > > Thanks
>> > > ZhangJian He
>> > >
>> > >
>> > > On Fri, 17 Feb 2023 at 23:24, ZhangJian He 
>> wrote:
>> > >
>> > > > Thank for StreamNative for willing to donate this project. This
>> means
>> > we
>> > > > don't have to develop and maintain a set of HTTP code from scratch.
>> My
>> > > idea
>> > > > aligns with Yunze's, and separating it into a standalone
>> > pulsar-admin-go
>> > > > project would be better. The **pulsarctl** repo contains bookkeeper
>> > http
>> > > > call too. Maybe we can have a project bookkeeper-admin-go ?（it's a
>> > liitle
>> > > > going off-topic ）
>> > > >
>> > > > Thanks
>> > > > ZhangJian He
>> > > >
>> > > >
>> > > > On Fri, 17 Feb 2023 at 20:29, PengHui Li 
>> > > wrote:
>> > > >
>> > > >> Hi Yunze,
>> > > >>
>> > > >> Yes, we can split it.
>> > > >> Both one repo with two modules or two repos works for me.
>> > > >>
>> > > >> The pulsarctl already have the admin API and CLI.
>> > > >> So I think we don’t need to develop another one.
>> > > >>
>> > > >> Best,
>> > > >> Penghui
>> > > >>
>> > > >> > On Feb 17, 2023, at 17:44, Yunze Xu > >
>> > > >> wrote:
>> > > >> >
>> > > >> > Hi PengHui,
>> > > >> >
>> > > >> > Now I changed my mind a bit. Even if the pulsarctl was
>> contributed
>> > to
>> > > >> > the Apache Foundation, I think we should also avoid adding it as
>> the
>> > > >> > dependency. What we need is an API layer but not the CLI, while
>> > > >> > pulsarctl couples the API and CLI.
>> > > >> >
>> > > >> > At the moment, my expectation is:
>> > > >> > 1. Use a separate repo (e.g. pulsar-admin-go) to implement the
>> admin
>> > > >> > APIs in Golang.
>> > > >> > 2. Depend this new repo in pulsarctl.
>> > > >> >
>> > > >> > Then we will have three Go projects:
>> > > >> > - pulsar-client-go: The Pulsar Go client APIs
>> > > >> > - pulsar-admin-go: The Pulsar Go admin APIs
>> > > >> > - pulsarctl: The admin CLI tool written in Go
>> > > >> >
>> > > >> > Thanks,
>> > > >> > Yunze
>> > > >> >
>> > > >> > On Fri, Feb 17, 2023 at 4:22 PM PengHui Li 
>> > > wrote:
>> > > >> >>
>> > > >> >> I checked with Sijie today.
>> > > >> >> StreamNative can contribute the pulsarctl project to Apache
>> > > Foundation.
>> > > >> >>
>> > > >> >> Regards,
>> > > >> >> Penghui
>> > > >> >>
>> > > >> >> On Fri, Feb 17, 2023 at 4:02 PM Enrico Olivelli <
>> > eolive...@gmail.com
>> > > >
>> > > >> wrote:
>> > > >> >>
>> > > >> >>> I agree to add an admin API to the go client, this would be
>> very
>> > > >> helpful.
>> > > >> >>>
>> > > >> >>> Il giorno ven 17 feb 2023 alle ore 08:44 Zixuan Liu
>> > > >> >>>  ha scritto:
>> > > >> 
>> > > >>  Hi Zhangjian,
>> > > >> 
>> > > >>  This is a good idea to write the admin client by golang, but I
>> > > don't
>> > > >>  suggest add the admin features to pulsar-go-client, it's
>> better
>> > to
>> > > >> use a
>> > > >>  new repository to do that to  separate dependencies.
>> > > >> 
>> > > >>  BTW, StreamNative has a pulsarctl [0] tool, which includes the
>> > > admin
>> > > >> api.
>> > > >> 
>> > > >> >> It's better to reuse existing code rather than reinventing
>> the
>> > > >> wheel.
>> > > >> 
>> > > >>  I aggred this point. If possible, we can integrate the
>> pulsarctl
>> > to
>> > > >> this
>> > > >>  new project.

Re: [VOTE] Pulsar Client Python Release 3.1.0 Candidate 3

[Yunze Xu]

Okay, I will check it soon.

Thanks,
Yunze

On Tue, Feb 28, 2023 at 5:17 PM Zike Yang  wrote:
>
> Hi, Yunze
>
> It raises an exception when I run the consumer example.
>
> Here are my environments:
> * macos 12.06 x86_64
> * python 3.7
>
> Here are my reproduce steps:
> * Start the pulsar standalone
> * Start the consumer example
> * Start the producer example
> * The consumer can receive messages successfully.
> * Stop the consumer, then it throws some exception:
> ```
> ➜  examples git:(main) ✗ python consumer.py
> 2023-02-28 17:11:04.742 INFO  [0x113b19600] Client:87 | Subscribing on
> Topic :my-topic
> 2023-02-28 17:11:04.742 INFO  [0x113b19600] ClientConnection:190 |
> [ -> pulsar://localhost:6650] Create ClientConnection,
> timeout=1
> 2023-02-28 17:11:04.742 INFO  [0x113b19600] ConnectionPool:97 |
> Created connection for pulsar://localhost:6650
> 2023-02-28 17:11:04.745 INFO  [0x73606000] ClientConnection:388 |
> [127.0.0.1:49258 -> 127.0.0.1:6650] Connected to broker
> 2023-02-28 17:11:04.857 INFO  [0x73606000] HandlerBase:72 |
> [persistent://public/default/my-topic, my-subscription, 0] Getting
> connection from pool
> 2023-02-28 17:11:04.874 INFO  [0x73606000] ConsumerImpl:238 |
> [persistent://public/default/my-topic, my-subscription, 0] Created
> consumer on broker [127.0.0.1:49258 -> 127.0.0.1:6650]
> Received message 'hello' id='(132,30,-1,0)'
> Received message 'hello' id='(132,31,-1,0)'
> Received message 'hello' id='(132,32,-1,0)'
> Received message 'hello' id='(132,33,-1,0)'
> Received message 'hello' id='(132,34,-1,0)'
> Received message 'hello' id='(132,35,-1,0)'
> Received message 'hello' id='(132,36,-1,0)'
> Received message 'hello' id='(132,37,-1,0)'
> Received message 'hello' id='(132,38,-1,0)'
> Received message 'hello' id='(132,39,-1,0)'
> ^CKeyboardInterrupt
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
>   File "consumer.py", line 32, in 
> msg = consumer.receive()
>   File 
> "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pulsar/__init__.py",
> line 1243, in receive
> msg = self._consumer.receive()
> SystemError
> 2023-02-28 17:11:07.367 WARN  [0x113b19600] ConsumerImpl:126 |
> [persistent://public/default/my-topic, my-subscription, 0] Destroyed
> consumer which was not properly closed
> 2023-02-28 17:11:07.367 INFO  [0x113b19600] ConsumerImpl:134 |
> [persistent://public/default/my-topic, my-subscription, 0] Closed
> consumer for race condition: 0
> libc++abi: terminating with uncaught exception of type
> std::__1::bad_weak_ptr: bad_weak_ptr
> [1]56483 abort  python consumer.py
> ```
> This causes the program to crash.
>
> I'm using the example codes here:
> https://github.com/apache/pulsar-client-python/tree/main/examples
>
> Could you take a look? Thanks.
>
> BR,
> Zike Yang
>
> On Tue, Feb 28, 2023 at 10:37 AM PengHui Li  wrote:
> >
> > +1 (binding)
> >
> > - install on macos
> > - start a standalone (latest master)
> > - run the example
> > https://github.com/apache/pulsar-client-python#running-examples
> >
> > Regards,
> > Penghui
> >
> > On Thu, Feb 23, 2023 at 11:00 PM Enrico Olivelli 
> > wrote:
> >
> > > Thank you Yunze for double checking.
> > >
> > > I don't have time to test the release, so I am voting +0 (and not -1)
> > >
> > > Enrico
> > >
> > > Il giorno gio 23 feb 2023 alle ore 12:57 Yunze Xu
> > >  ha scritto:
> > > >
> > > > Hi Enrico,
> > > >
> > > > I will test more operation systems and open a discussion soon. For
> > > > now, I just tested the TLS encryption and token authentication, and I
> > > > found this issue does not exist as expected for both Python and
> > > > Node.js clients.
> > > > - Windows: Only Python client works
> > > > - Ubuntu: Both work
> > > > - macOS: Not tested yet
> > > >
> > > > But the OAuth2 authentication case doesn't work. It's caused by
> > > > https://github.com/apache/pulsar/pull/16064 and fixed by
> > > > https://github.com/apache/pulsar-client-cpp/pull/190.
> > > >
> > > > When the protocol of the issuer URL is HTTPS:
> > > > - Before #16064, OAuth2 authentication works by skipping verifying the
> > > > peer, it's dangerous for security reasons
> > > > - After #16064 and before #190, there is no way to perform OAuth2
> > > authentication
> > > > - After #190, users can configure the `tls_trust_certs_file_path` to do
> > > that.
> > > >
> > > > So I think it should not be treated as a regression and should not
> > > > block this release. We can support automatically detecting the CA
> > > > certs for OAuth2 authentication later.
> > > >
> > > > The regression should be the case when the TLS encryption is enabled.
> > > > And as I've said at the beginning, I'm going to do more tests and open
> > > > a discussion.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Thu, Feb 23, 2023 at 6:06 PM Enrico Olivelli 
> > > wrote:
> > > > >
> > > > > Does this version have the same problems about 

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 2

[Nozomi Kurihara]

+1 (binding)

* checked license headers
* verified checksum and signature
* install from npm and run producer/consumer

Thanks,
Nozomi

2023年2月26日(日) 12:23 Baodi Shi :

> Hi everyone,
>
> This is the first release candidate for Apache Pulsar Node.js client,
> version 1.8.1.
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.1+is%3Aclosed
>
> Please download the source files and review this release candidate:
> - Download the source package, verify shasum and asc
> - Follow the README.md to build and run the Pulsar Node.js client.
>
> The release candidate package has been published to the npm registry:
> https://www.npmjs.com/package/pulsar-client/v/1.8.1-rc.2
> You can install it by `npm i pulsar-client@1.8.1-rc.2
> --pulsar_binary_host_mirror=
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>  and
> verify the package.
>
> You can refer to this repository to verify tls related features:
>
>- https://github.com/shibd/pulsar-client-tls-test
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Source files:
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.1-rc.2/
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> SHA-512 checksum:
>
>
> e596fef3eba6fbd25413ccf6eee3cf0a22c24625ff699b4f6d49676ebe2a053f4864ecdee79eb4dbde4fde143e867ec5c1fe667d0a1db07370b9d2abdb806ac3
>  apache-pulsar-client-node-1.8.1.tar.gz
>
> The tag to be voted upon:
> v1.8.1-rc.2(f0a5e0b)
> https://github.com/apache/pulsar-client-node/releases/tag/v1.8.1-rc.2
>
> Please review and vote on the release candidate #1 for the version 1.8.1,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
>
> Thanks,
> Baodi Shi
>