Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Zike Yang]

+1 (non-binding)

Thanks,
Zike Yang

On Wed, Feb 22, 2023 at 10:12 PM Jiuming Tao
 wrote:
>
> It makes sense
>
> > 2023年2月20日 17:32，Zike Yang  写道：
> >
> > Hi, Jiuming,
> >
> > Overall looks good to me. Left some comments for the Compatibility section:
> >
> >> In the next release after 2.11.1, if users want to create Durable 
> >> subscriptions on NonPersistentTopic, will throw an exception.
> >
> > I think this is an improvement but not a bug fix. Right?
> > The next feature release is 3.0.0. I think we should print warn logs
> > in 3.0.0 and throw exceptions may be in 3.1.0. Otherwise, it will
> > bring the breaking change here in 3.0.0.
> > The 2.11.1 is a patch release. I don't recommend cherry-picking this
> > PIP to branch-2.11 because it's not a critical bug fix.
> >
> > BR,
> > Zike Yang
> >
> >
> > Zike Yang
> >
> > On Fri, Feb 17, 2023 at 12:52 AM Baodi Shi  wrote:
> >>
> >> +1 (non-binding)
> >>
> >> Thanks,
> >> Baodi Shi
> >>
> >>
> >> 在 2023年2月16日 21:44:17 上，Asaf Mesika  写道：
> >>
> >>> +1 (non-binding)
> >>>
> >>>
> >>> On Thu, Feb 16, 2023 at 11:11 AM Jiuming Tao 
> >>>  
> >>> wrote:
> >>>
> >>>
> >>> I’ve added the `Compatibility` selection into the PIP, please help review
> >>>
> >>> and vote the PIP
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Tao Jiuming
> >>>
> >>>
> >>>
> >>>
>  2023年2月15日 14:58，Zike Yang  写道：
> >>>
> 
> >>>
>  Hi, Jiuming
> >>>
> 
> >>>
> > bump
> >>>
> 
> >>>
>  As for the discussion here[0], could you add a `Compatibility` section
> >>>
>  to talk about compatibility in more detail? WDYT?
> >>>
>  Then we could start the vote again.
> >>>
> 
> >>>
>  [0] https://lists.apache.org/thread/2bjg39zh7z38bzbnqngbo5l4jzkjttrq
> >>>
> 
> >>>
>  Thanks,
> >>>
>  Zike Yang
> >>>
> 
> >>>
>  On Wed, Feb 15, 2023 at 1:34 PM Tao Jiuming  wrote:
> >>>
> >
> >>>
> >
> >>>
> > bump
> >>>
> >
> >>>
> > On 2023/02/13 06:56:09 Jiuming Tao wrote:
> >>>
> >> Hi all,
> >>>
> >>
> >>>
> >> I would like to start a VOTE on `PIP-245: Make subscriptions of
> >>>
> >>> non-persistent topic non-durable`.
> >>>
> >>
> >>>
> >> Motivation:
> >>>
> >>
> >>>
> >> There are two types of subscriptions for a topic: Durable and
> >>>
> >>> Non-durable.
> >>>
> >>
> >>>
> >> We create a Consumer with a Durable subscription and a Reader with a
> >>>
> >>> Non-durable subscription.
> >>>
> >>
> >>>
> >> But for NonPersistentTopic, creating a Durable subscription is
> >>>
> >>> meaningless, NonPersistentSubscription doesn't have a ManagedCursor to
> >>>
> >>> persistent its data. After its consumer disconnected, the subscription
> >>>
> >>> couldn't be removed automatically if we didn't set the value of
> >>>
> >>> subscriptionExpirationTimeMinutes greater than 0.
> >>>
> >>
> >>>
> >> For subscriptionExpirationTimeMinutes, it controls the subscription
> >>>
> >>> expiration of NonPersistentTopic and PersistentTopic, if we set the value
> >>>
> >>> of subscriptionExpirationTimeMinutes greater than 0, it may lead to data
> >>>
> >>> loss(The durable subscriptions of PersistentTopic also can be removed).
> >>>
> >>
> >>>
> >> And the Non-durable subscriptions will be removed automatically after
> >>>
> >>> all the consumers disconnected, it's the existing logic.
> >>>
> >>
> >>>
> >> For the purpose of removing the subscriptions which have no active
> >>>
> >>> consumers of NonPersistentTopic and the above reasons, we can make all the
> >>>
> >>> subscriptions of a NonPersistentTopic Non-durable.
> >>>
> >>
> >>>
> >>
> >>>
> >>
> >>>
> >> For more details, you can read:
> >>>
> >>> https://github.com/apache/pulsar/issues/19448 <
> >>>
> >>> https://github.com/apache/pulsar/issues/19448>
> >>>
> >>
> >>>
> >> And the discuss thread is available at:
> >>>
> >>> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst <
> >>>
> >>> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst>
> >>>
> >>
> >>>
> >> Thanks,
> >>>
> >> Tao Jiuming
> >>>
> >>>
> >>>
> >>>
>

Re: [DISCUSS] Using bouncycastle fips instead bouncycastle non-fips

[Zixuan Liu]

Thanks @Michael for the context!

> 2020-10-28 01:53:06 UTC - jia zhai: we could consider that in the future.
current way is to align with the old manner.  usually user will provide
their own security jars, such as BouncyCastle, and non-fips version maybe
used in a more wide range.

That's valuable, but if we use the BouncyCastle stuff + FIPS compliance, it
would be better(From a safety point of view). I will try to communicate
with jia offline.

> Also, do we need to update our cryptography notice on our README about
our bouncy castle usage [4] if we make this the default?

Sure, we should update this notice. Right now we are using the `Conscrypt`
and `netty-tcnative` to provide the SSL.

Thanks,
Zixuan

Michael Marshall  于2023年2月23日周四 04:11写道：

> I am +1, but I am not familiar with the nuanced differences of these
> jars, so it'd be valuable to get other opinions, too.
>
> I tried to understand why the default is the way it is, and it looks
> like Jia Zhai was involved in the initial work [0].
>
> I see in the slack digest on our mailing list that there is an
> indication that we could make the FIPS version the default. Here are
> some relevant messages from the ML [1][2][3]
>
> > 2020-10-26 06:43:48 UTC - Anup Ghatage: I noticed that Pulsar doesn’t
> have BouncyCastle FIPS switched on by default but the documentation leaves
> it up to the implementers to choose.
> Is there any specific reason for this? I thought BC-FIPS comes with
> the usual BouncyCastle stuff + FIPS compliance (which would be a good
> thing) right?
> > 2020-10-26 08:37:13 UTC - Sijie Guo: @jia zhai would have a better
> answer for it.
> > 2020-10-26 12:59:16 UTC - jia zhai: @Sijie Guo @Anup Ghatage right,
> there is no specific reason for this. It seems be more related to the
> project building,  we may need to config the profile to build different bc
> type. but this seems a little hard in maven:joy:.
> It would be helpful, if anyone would like to contribute to this feature.
> > 2020-10-26 16:31:57 UTC - Anup Ghatage: I’m thinking why not keep it on
> by default? There aren’t any API differences for the most part. That way
> Pulsar will be FIPS compliant all the time.
> > 2020-10-28 01:53:06 UTC - jia zhai: we could consider that in the
> future. current way is to align with the old manner.  usually user will
> provide their own security jars, such as BouncyCastle, and non-fips version
> maybe used in a more wide range.
>
> Also, do we need to update our cryptography notice on our README about
> our bouncy castle usage [4] if we make this the default?
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/6588
> [1] https://lists.apache.org/thread/fln8o94t0gxnd54fr7tn4hrjp23mj48r
> [2] https://lists.apache.org/thread/xmrhyo1fkdhm4l9xz0t66yk5pk5g5f6p
> [3] https://lists.apache.org/thread/fs8rx620oq7q7px1mqs3k7qdoz3oz0s4
> [4] https://github.com/apache/pulsar#crypto-notice
>
> On Wed, Feb 22, 2023 at 7:56 AM Zixuan Liu  wrote:
> >
> > > 1. What is FIPS?
> >
> > FIPS (Federal Information Processing Standards) are a set of standards
> that
> > describe document processing, encryption algorithms and other information
> > technology standards for use within non-military government agencies and
> by
> > government contractors and vendors who work with the agencies.
> >
> > > 2. Why is the FIPS version safer exactly?
> >
> > FIPS standard is strict. When using the FIPS version, this is also very
> > strict and standard.
> >
> > > 3. What is bouncycastle used exactly in Pulsar?
> >
> > We use the bouncycastle as the TLS provider,  and used for the end-to-end
> > message encryption.
> >
> > Thanks,
> > Zixuan
> >
> > Asaf Mesika  于2023年2月22日周三 21:23写道：
> >
> > > Can you elaborate a bit:
> > > 1. What is FIPS?
> > > 2. Why is the FIPS version safer exactly?
> > > 3. What is bouncycastle used exactly in Pulsar?
> > >
> > >
> > >
> > > On Wed, Feb 22, 2023 at 11:58 AM Zixuan Liu  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to discuss using the bouncycastle fips instead of the
> > > > bouncycastle non-fips.
> > > >
> > > > The bouncycastle is a Java library that complements the default Java
> > > > Cryptographic Extension (JCE), which has two versions: fips version
> and
> > > > non-fips version.
> > > >
> > > > The fips version is safer than non-fips. When the security level is
> very
> > > > high, many policies require the fips version, but the Pulsar default
> uses
> > > > the non-fips version. Switch this is complex, because
> > > > the `pulsar-client-messagecrypto-bc` module and root project depends
> on
> > > the
> > > > non-fips, so I suggest we switch to fips version from non-fips.
> > > >
> > > > Reference:
> > > > - https://www.bouncycastle.org/
> > > > - https://www.bouncycastle.org/fips_faq.html
> > > > -
> https://en.wikipedia.org/wiki/Federal_Information_Processing_Standards
> > > >
> > > > Thanks,
> > > > Zixuan
> > > >
> > >
>

Re: [VOTE] Pulsar Client Python Release 3.1.0 Candidate 3

[Matteo Merli]

+1 bindind
--
Matteo Merli



On Fri, Feb 17, 2023 at 3:27 AM Yunze Xu 
wrote:

> This is the 3rd release candidate for Apache Pulsar Client Python,
> version 3.1.0.
>
> It fixes the following issues:
> https://github.com/apache/pulsar-client-python/milestone/2?closed=1
>
> *** Please download, test and vote on this release. This vote will
> stay open for at least 72 hours ***
>
> Python wheels:
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-3.1.0-candidate-3/
>
> The supported python versions are 3.7, 3.8, 3.9, 3.10 and 3.11. The
> supported platforms and architectures are:
> - Windows x86_64 (windows/)
> - glibc-based Linux x86_64 (linux-glibc-x86_64/)
> - glibc-based Linux arm64 (linux-glibc-arm64/)
> - musl-based Linux x86_64 (linux-musl-x86_64/)
> - musl-based Linux arm64 (linux-musl-arm64/)
> - macOS universal 2 (macos/)
>
> You can download the wheel (the `.whl` file) according to your own OS
> and Python version
> and install the wheel:
> - Windows: `py -m pip install *.whl --force-reinstall`
> - Linux or macOS: `python3 -m pip install *.whl --force-reinstall`
>
> The tag to be voted upon: v3.1.0-candidate-3
> (9ed92ecee632c42b81a3198b8824d70d080af7f0)
>
> https://github.com/apache/pulsar-client-python/releases/tag/v3.1.0-candidate-3
>
> Pulsar's KEYS file containing PGP keys you use to sign the release:
> https://dist.apache.org/repos/dist/release/pulsar/KEYS
>
> Please download the Python wheels and follow the README to test.
>

Re: [DISCUSS] Using bouncycastle fips instead bouncycastle non-fips

[Michael Marshall]

I am +1, but I am not familiar with the nuanced differences of these
jars, so it'd be valuable to get other opinions, too.

I tried to understand why the default is the way it is, and it looks
like Jia Zhai was involved in the initial work [0].

I see in the slack digest on our mailing list that there is an
indication that we could make the FIPS version the default. Here are
some relevant messages from the ML [1][2][3]

> 2020-10-26 06:43:48 UTC - Anup Ghatage: I noticed that Pulsar doesn’t have 
> BouncyCastle FIPS switched on by default but the documentation leaves it up 
> to the implementers to choose.
Is there any specific reason for this? I thought BC-FIPS comes with
the usual BouncyCastle stuff + FIPS compliance (which would be a good
thing) right?
> 2020-10-26 08:37:13 UTC - Sijie Guo: @jia zhai would have a better answer for 
> it.
> 2020-10-26 12:59:16 UTC - jia zhai: @Sijie Guo @Anup Ghatage right, there is 
> no specific reason for this. It seems be more related to the project 
> building,  we may need to config the profile to build different bc type. but 
> this seems a little hard in maven:joy:.
It would be helpful, if anyone would like to contribute to this feature.
> 2020-10-26 16:31:57 UTC - Anup Ghatage: I’m thinking why not keep it on by 
> default? There aren’t any API differences for the most part. That way Pulsar 
> will be FIPS compliant all the time.
> 2020-10-28 01:53:06 UTC - jia zhai: we could consider that in the future. 
> current way is to align with the old manner.  usually user will provide their 
> own security jars, such as BouncyCastle, and non-fips version maybe used in a 
> more wide range.

Also, do we need to update our cryptography notice on our README about
our bouncy castle usage [4] if we make this the default?

Thanks,
Michael

[0] https://github.com/apache/pulsar/pull/6588
[1] https://lists.apache.org/thread/fln8o94t0gxnd54fr7tn4hrjp23mj48r
[2] https://lists.apache.org/thread/xmrhyo1fkdhm4l9xz0t66yk5pk5g5f6p
[3] https://lists.apache.org/thread/fs8rx620oq7q7px1mqs3k7qdoz3oz0s4
[4] https://github.com/apache/pulsar#crypto-notice

On Wed, Feb 22, 2023 at 7:56 AM Zixuan Liu  wrote:
>
> > 1. What is FIPS?
>
> FIPS (Federal Information Processing Standards) are a set of standards that
> describe document processing, encryption algorithms and other information
> technology standards for use within non-military government agencies and by
> government contractors and vendors who work with the agencies.
>
> > 2. Why is the FIPS version safer exactly?
>
> FIPS standard is strict. When using the FIPS version, this is also very
> strict and standard.
>
> > 3. What is bouncycastle used exactly in Pulsar?
>
> We use the bouncycastle as the TLS provider,  and used for the end-to-end
> message encryption.
>
> Thanks,
> Zixuan
>
> Asaf Mesika  于2023年2月22日周三 21:23写道：
>
> > Can you elaborate a bit:
> > 1. What is FIPS?
> > 2. Why is the FIPS version safer exactly?
> > 3. What is bouncycastle used exactly in Pulsar?
> >
> >
> >
> > On Wed, Feb 22, 2023 at 11:58 AM Zixuan Liu  wrote:
> >
> > > Hi all,
> > >
> > > I would like to discuss using the bouncycastle fips instead of the
> > > bouncycastle non-fips.
> > >
> > > The bouncycastle is a Java library that complements the default Java
> > > Cryptographic Extension (JCE), which has two versions: fips version and
> > > non-fips version.
> > >
> > > The fips version is safer than non-fips. When the security level is very
> > > high, many policies require the fips version, but the Pulsar default uses
> > > the non-fips version. Switch this is complex, because
> > > the `pulsar-client-messagecrypto-bc` module and root project depends on
> > the
> > > non-fips, so I suggest we switch to fips version from non-fips.
> > >
> > > Reference:
> > > - https://www.bouncycastle.org/
> > > - https://www.bouncycastle.org/fips_faq.html
> > > - https://en.wikipedia.org/wiki/Federal_Information_Processing_Standards
> > >
> > > Thanks,
> > > Zixuan
> > >
> >

Re: [VOTE] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[Alexander Preuss]

I would like to conclude the vote, there are 10 votes on the ML+GitHub of
which 3 were binding.

Best,
Alexander

On Thu, Feb 9, 2023 at 10:27 AM PengHui Li  wrote:

> +1 (binding)
>
> Penghui
>
> > On Feb 9, 2023, at 17:26, Nicolò Boschi  wrote:
> >
> > +1 binding
> > Nicolò
> >
> > Il giorno gio 9 feb 2023 alle 04:48 Zike Yang  ha
> scritto:
> >
> >> +1 (non-binding)
> >>
> >> Thanks,
> >> Zike Yang
> >>
> >> On Wed, Feb 8, 2023 at 11:08 PM Baodi Shi  wrote:
> >>>
> >>> +1 (non-binding)
> >>>
> >>>
> >>> Thanks,
> >>> Baodi Shi
> >>>
> >>>
> >>> 在 2023年2月8日 19:12:41 上，r...@apache.org  写道：
> >>>
>  +1(non-binding)
> 
>  --
>  Thanks
>  Xiaolong Ran
> 
>  Michael Marshall  于2023年2月8日周三 12:22写道：
> 
>  +1 (binding)
> 
> 
>  Thanks,
> 
>  Michael
> 
> 
>  On Tue, Feb 7, 2023 at 7:00 AM Rui Fu  wrote:
> 
> >
> 
> > +1
> 
> >
> 
> > Best,
> 
> >
> 
> > Rui Fu
> 
> > On Jan 30, 2023 at 19:12 +0800, Alexander Preuss
> 
>  , wrote:
> 
> >> Now that the DISCUSSION thread has received some more input and
> >> some
> 
>  time
> 
> >> has passed I would like to start the VOTE again.
> 
> >>
> 
> >> Best,
> 
> >> Alexander
> 
> >>
> 
> >> On Tue, Jan 17, 2023 at 3:21 AM Bonan Hou <
> >> bonan@streamnative.io
> 
>  .invalid>
> 
> >> wrote:
> 
> >>
> 
> >>> +1
> 
> >>>
> 
> >>> Bonan Hou
> 
> >>>
> 
> >>> On Mon, Jan 16, 2023 at 9:56 PM Alexander Preuss
> 
> >>>  wrote:
> 
> >>>
> 
>  Hi everyone,
> 
> 
> 
>  I'm starting the VOTE for PIP-237: Make PulsarAdmin accessible
> >> in
> 
>  SinkContext and SourceContext:
> 
>  https://github.com/apache/pulsar/issues/19123
> 
> 
> 
>  Discussion thread:
> 
> 
> >> https://lists.apache.org/thread/b8r6b4p87qzfgfwmqfql7mqx8kmgorxg
> 
> 
> 
>  I'd like to go ahead with implementing option B presented in
> >> the
> 
>  PIP
> 
>  description on GitHub.
> 
> 
> 
>  Thanks,
> 
>  Alexander
> 
> 
> 
> >>>
> 
> 
> 
> >>
> > --
> > Nicolò Boschi
>
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

[Yunze Xu]

I just noticed the `AuthenticationToken` is used here, so it's not
related to #16064. The root cause is only that the prebuilt C++ add-on
might have a different CA cert store path with the local env.

Thanks,
Yunze

On Wed, Feb 22, 2023 at 10:30 PM Yunze Xu  wrote:
>
> Hi Enrico and Nicolo,
>
> The Node.js client is based on the C++ client so it requires the C++ Add-on. 
> [1]
>
> For 1.7.0 or earlier, when users install the client, the `npm install
> pulsar-client` command builds the add-on. It means they have to
> install the C++ compiler. For 1.8.0 or later, we have provided the
> prebuilt C++ add-on. However, the prebuilt C++ add-on depends on
> libcurl, whose CA cert store path is determined when building the
> libcurl.
>
> Before 1.8.0, the Node.js client depends on C++ client 2.x.y, which
> does not include #16064 [2] , which fixes the CVE. Before #16064, the
> libcurl does not verify the peer, so it would lead to the CVE.
>
> ```c++
> curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
> ```
>
> You can find a more detailed explanation here [3]. In short, when
> users install the Node.js client using the prebuilt C++ add-on, the CA
> cert store path is fixed but it could be different in a different OS.
> So we have to specify the CA cert store path via the
> tlsTrustCertsFilePath config now.
>
> To prevent the code changes, maybe we have to fallback to the old ways
> to install the Node.js client, i.e. building the C++ add-on in users'
> local env.
>
> [1] https://nodejs.org/api/addons.html#c-addons
> [2] https://github.com/apache/pulsar/pull/16064
> [3] 
> https://github.com/apache/pulsar-client-cpp/issues/184#issuecomment-1420509146
>
>
> Thanks,
> Yunze
>
> On Wed, Feb 22, 2023 at 6:10 PM Enrico Olivelli  wrote:
> >
> > The problem reported by Nicolò seems a real show stopper
> > -1 (binding)
> >
> > Is there any volunteer on fixing the problem ?
> >
> > I think that most of Production users use TLS and this problem prevent
> > anyone to upgrade from 1.7.x to 1.8.x
> >
> > Enrico
> >
> > Il giorno mer 22 feb 2023 alle ore 10:33 Nicolò Boschi
> >  ha scritto:
> > >
> > > Yes, I'm on MacOS
> > >
> > > Adding the explicit path to the system CA worked, thanks.
> > >
> > > I wonder why it wasn't needed in 1.7.0 and now it's required in 1.8.1.
> > > This can be problematic for users because the client upgrade requires code
> > > changes and redeployment.
> > > Maybe it's a regression from 1.8.0, but I think that we need to get that
> > > fixed in 1.8.1
> > >
> > > Nicolò Boschi
> > >
> > >
> > > Il giorno mer 22 feb 2023 alle ore 10:17 Baodi Shi  ha
> > > scritto:
> > >
> > > > Hi, @Nicolò Boschi 
> > > >
> > > > Is your system macOS(arm64)?
> > > >
> > > > Maybe you should set tlsTrustCertsFilePath:
> > > >
> > > >   const client = new Pulsar.Client({
> > > > tlsTrustCertsFilePath:'/etc/ssl/cert.pem',
> > > >   });
> > > >
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Baodi Shi
> > > >
> > > >
> > > > 在 2023年2月22日 15:59:28 上，Nicolò Boschi  写道：
> > > >
> > > >> Hi,
> > > >>
> > > >> I'm having issues while validating the fix related to the hostname
> > > >> verification: https://github.com/apache/pulsar-client-cpp/pull/126
> > > >> My usecase is with a valid TLS certificate signed by a CA (not a
> > > >> self-signed one).
> > > >>
> > > >> My code is very simple (see below): it creates a client with token 
> > > >> auth +
> > > >> TLS and sends some messages.
> > > >>
> > > >> It works well with node client 1.7.0 with cpp client 3.1.2
> > > >> It fails with node client 1.8.0 (as expected)
> > > >> It still fails with the rc: 1.8.1-rc.1
> > > >>
> > > >> (I'm installing the dependency with "npm i pulsar-client@1.8.1-rc.1
> > > >> --pulsar_binary_host_mirror=
> > > >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/;
> > > >>
> > > >> The error I'm seeing is this one:
> > > >>
> > > >> [INFO][ClientConnection:388] Connected to broker
> > > >> [ERROR][ClientConnection:488] Handshake failed: certificate verify 
> > > >> failed
> > > >> (SSL routines, tls_process_server_certificate)
> > > >> [INFO][ClientConnection:1600] Connection closed with ConnectError
> > > >>
> > > >> Note that setting `tlsValidateHostname: true` "resolves" the problem,
> > > >> however it's not acceptable as you know.
> > > >>
> > > >> I'm pretty sure that it's related to the cpp client dependency, however
> > > >> I'm
> > > >> not very familiar with it and how it's bundled in the node client >= 
> > > >> 1.8.0
> > > >> Is there a way to verify if the bundled cpp client is actually the
> > > >> expected one?
> > > >>
> > > >>
> > > >>
> > > >> This is the code snippet:
> > > >> ```
> > > >> const tokenStr = asToken;
> > > >>  const pulsarUri = pulsarUrl;
> > > >>  const topicName = asTopic;
> > > >>
> > > >>  const auth = new Pulsar.AuthenticationToken({ token: tokenStr });
> > > >>  const client = new Pulsar.Client({
> > > >>serviceUrl: pulsarUri,
> > > >>authentication: auth,
> > > >>

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[SiNan Liu]

Sorry, my mistake. I removed the code and described the design to improve
the PROTOBUF_NATIVE schema compatibility checks. You can have a look. 

Asaf Mesika  于2023年2月22日周三 21:16写道：

> I read it but you're almost directly diving into the code - it will take me
> hours just to reverse engineer your design.
>
> Can you please include a "High Level Design" section in which you explain
> how you plan to tackle any issue?
> If I can read that section and explain to someone else how this will work,
> it means the section is complete.
>
> Let's leave the code to the PRs.
>
>
> On Sun, Feb 19, 2023 at 2:59 PM SiNan Liu  wrote:
>
> > Hi all,
> >
> > I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.
> >
> > We can talk about the current design here. Especially for the field type
> > change check rules, please give your valuable advice.
> >
> > Thanks,
> > Sinan
> >
>

[Discuss] PIP-248: Add backlog eviction metric

[太上玄元道君]

Hi all,

I've started a PIP to discuss: PIP-248 Add backlog eviction metric

### Motivation:

Pulsar has 2 configurations for the backlog eviction:
`backlogQuotaDefaultLimitBytes` and `backlogQuotaDefaultLimitSecond`, if
topic backlog reaches the threshold of any item, backlog eviction will be
triggered.

Before backlog eviction happens, we don't have a metric to monitor how long
that it can reaches the threshold.

We can provide a progress bar metric to tell users some topics is about to
trigger backlog eviction. And users can subscribe the alert to schedule
consumers.

For more details, please read the PIP at
https://github.com/apache/pulsar/issues/19601

Thanks,
Tao Jiuming

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

[Yunze Xu]

Hi Enrico and Nicolo,

The Node.js client is based on the C++ client so it requires the C++ Add-on. [1]

For 1.7.0 or earlier, when users install the client, the `npm install
pulsar-client` command builds the add-on. It means they have to
install the C++ compiler. For 1.8.0 or later, we have provided the
prebuilt C++ add-on. However, the prebuilt C++ add-on depends on
libcurl, whose CA cert store path is determined when building the
libcurl.

Before 1.8.0, the Node.js client depends on C++ client 2.x.y, which
does not include #16064 [2] , which fixes the CVE. Before #16064, the
libcurl does not verify the peer, so it would lead to the CVE.

```c++
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
```

You can find a more detailed explanation here [3]. In short, when
users install the Node.js client using the prebuilt C++ add-on, the CA
cert store path is fixed but it could be different in a different OS.
So we have to specify the CA cert store path via the
tlsTrustCertsFilePath config now.

To prevent the code changes, maybe we have to fallback to the old ways
to install the Node.js client, i.e. building the C++ add-on in users'
local env.

[1] https://nodejs.org/api/addons.html#c-addons
[2] https://github.com/apache/pulsar/pull/16064
[3] 
https://github.com/apache/pulsar-client-cpp/issues/184#issuecomment-1420509146


Thanks,
Yunze

On Wed, Feb 22, 2023 at 6:10 PM Enrico Olivelli  wrote:
>
> The problem reported by Nicolò seems a real show stopper
> -1 (binding)
>
> Is there any volunteer on fixing the problem ?
>
> I think that most of Production users use TLS and this problem prevent
> anyone to upgrade from 1.7.x to 1.8.x
>
> Enrico
>
> Il giorno mer 22 feb 2023 alle ore 10:33 Nicolò Boschi
>  ha scritto:
> >
> > Yes, I'm on MacOS
> >
> > Adding the explicit path to the system CA worked, thanks.
> >
> > I wonder why it wasn't needed in 1.7.0 and now it's required in 1.8.1.
> > This can be problematic for users because the client upgrade requires code
> > changes and redeployment.
> > Maybe it's a regression from 1.8.0, but I think that we need to get that
> > fixed in 1.8.1
> >
> > Nicolò Boschi
> >
> >
> > Il giorno mer 22 feb 2023 alle ore 10:17 Baodi Shi  ha
> > scritto:
> >
> > > Hi, @Nicolò Boschi 
> > >
> > > Is your system macOS(arm64)?
> > >
> > > Maybe you should set tlsTrustCertsFilePath:
> > >
> > >   const client = new Pulsar.Client({
> > > tlsTrustCertsFilePath:'/etc/ssl/cert.pem',
> > >   });
> > >
> > >
> > >
> > >
> > > Thanks,
> > > Baodi Shi
> > >
> > >
> > > 在 2023年2月22日 15:59:28 上，Nicolò Boschi  写道：
> > >
> > >> Hi,
> > >>
> > >> I'm having issues while validating the fix related to the hostname
> > >> verification: https://github.com/apache/pulsar-client-cpp/pull/126
> > >> My usecase is with a valid TLS certificate signed by a CA (not a
> > >> self-signed one).
> > >>
> > >> My code is very simple (see below): it creates a client with token auth +
> > >> TLS and sends some messages.
> > >>
> > >> It works well with node client 1.7.0 with cpp client 3.1.2
> > >> It fails with node client 1.8.0 (as expected)
> > >> It still fails with the rc: 1.8.1-rc.1
> > >>
> > >> (I'm installing the dependency with "npm i pulsar-client@1.8.1-rc.1
> > >> --pulsar_binary_host_mirror=
> > >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/;
> > >>
> > >> The error I'm seeing is this one:
> > >>
> > >> [INFO][ClientConnection:388] Connected to broker
> > >> [ERROR][ClientConnection:488] Handshake failed: certificate verify failed
> > >> (SSL routines, tls_process_server_certificate)
> > >> [INFO][ClientConnection:1600] Connection closed with ConnectError
> > >>
> > >> Note that setting `tlsValidateHostname: true` "resolves" the problem,
> > >> however it's not acceptable as you know.
> > >>
> > >> I'm pretty sure that it's related to the cpp client dependency, however
> > >> I'm
> > >> not very familiar with it and how it's bundled in the node client >= 
> > >> 1.8.0
> > >> Is there a way to verify if the bundled cpp client is actually the
> > >> expected one?
> > >>
> > >>
> > >>
> > >> This is the code snippet:
> > >> ```
> > >> const tokenStr = asToken;
> > >>  const pulsarUri = pulsarUrl;
> > >>  const topicName = asTopic;
> > >>
> > >>  const auth = new Pulsar.AuthenticationToken({ token: tokenStr });
> > >>  const client = new Pulsar.Client({
> > >>serviceUrl: pulsarUri,
> > >>authentication: auth,
> > >>operationTimeoutSeconds: 30,
> > >>tlsCertificateFilePath: "",
> > >>tlsValidateHostname: false
> > >>  });
> > >>  Pulsar.Client.setLogHandler((level, file, line, message) => {
> > >>console.log('[%s][%s:%d] %s', Pulsar.LogLevel.toString(level), file,
> > >> line, message);
> > >>  });
> > >>
> > >>  const producer = await client.createProducer({
> > >>topic: topicName,
> > >>  })
> > >>
> > >>  for (let i = 0; i < 10; i += 1) {
> > >>await producer.send({
> > >>  data: Buffer.from("nodejs-message-" + i),
> > >>});
> > >>

Re: Does anyone build UI for Pulsar?

[Enrico Olivelli]

Asaf,

Il giorno mer 22 feb 2023 alle ore 14:19 Asaf Mesika
 ha scritto:
>
> Wondering out loud if there are generic UI for message brokers?

Unfortunately the only "standard" API to access Messaging services is
JMS but it is pretty limited
as it allows only to produce and consume.

Also a Pulsar UI should be able to allow the user to manage
everything, and Pulsar has many cool features that are not available
to many other systems, especially builtin multi-tenancy.

Enrico

>
> On Tue, Feb 21, 2023 at 6:27 PM Kiryl Valkovich 
> wrote:
>
> > Devin,
> >
> > The most desired feature I need as an event-driven application developer
> > is the ability to read and filter messages in topics.
> > With the NonDurable subscription type, it shouldn’t affect the topic’s
> > data.
> >
> > The ability to easily produce a message or generate a bunch of them, based
> > on some rules, allows me to quickly check how the application behaves in
> > different scenarios.
> >
> > It’s also useful for the QA audience.
> >
> > From: Devin Bost 
> > Date: Tuesday, February 21, 2023 at 4:58 PM
> > To: Dev 
> > Subject: Re: Does anyone build UI for Pulsar?
> > Kiryl,
> >
> > > Just make a side-by-side comparison of Pulsar Manager with any of the
> > following options:
> > > - Conduktor (Commercial).
> > > - Kafka UI by Provectus (Apache License 2.0).
> > > - Redpanda Console (ex-Kowl) (Mixed license).
> > >
> > > I see a significant gap in user experience between any of them and the
> > Pulsar Manager.
> >
> > Thanks for bringing up this subject. You're making good points that we have
> > a gap in this area.
> >
> > In your perspective, where do you think the biggest gaps are in terms of
> > functionality?
> >
> > --
> > Devin Bost
> > Sent from mobile
> > Cell: 801-400-4602
> >
> > On Mon, Feb 20, 2023, 8:54 AM Enrico Olivelli  wrote:
> >
> > > Il giorno lun 20 feb 2023 alle ore 14:41 Kiryl Valkovich
> > >  ha scritto:
> > > >
> > > > Enrico, it’s easy.
> > > >
> > > > When I tried it, the basic functionality just didn’t work.
> > > >
> > > > Just make a side-by-side comparison of Pulsar Manager with any of the
> > > following options:
> > > > - Conduktor (Commercial).
> > > > - Kafka UI by Provectus (Apache License 2.0).
> > > > - Redpanda Console (ex-Kowl) (Mixed license).
> > > >
> > > > I see a significant gap in user experience between any of them and the
> > > Pulsar Manager.
> > > > Also, I just don’t see much contribution activity.
> > > >
> > > > Redpanda Console and Kafka UI, each have about x10 times more commits
> > in
> > > a year, than Pulsar Manager has in the last 3 years. Therefore I can’t
> > > conclude that the project is alive.
> > >
> > > (Unfortunately) I 100% agree with you.
> > >
> > > >
> > > > Regarding the “Why from scratch?” question:
> > > > - I found it easier for me personally because of the different tech
> > > stack I use.
> > > > - Starting from scratch is a good opportunity to learn Pulsar itself.
> > > > - I’d want to make a product around that.
> > > >
> > > > I'm not claiming that my result will be 100% better, but why not try?
> > >
> > > I understand your points.
> > > I hope you will share your project with OSS license, this way it will
> > > be easier for Pulsar users to try it out and contribute.
> > > >
> > > > If someone is already doing or plans to do similar work, let’s
> > > collaborate!
> > >
> > > Thanks for sharing
> > >
> > > Enrico
> > >
> > > >
> > > > From: Enrico Olivelli 
> > > > Date: Monday, February 20, 2023 at 2:04 PM
> > > > To: dev@pulsar.apache.org 
> > > > Subject: Re: Does anyone build UI for Pulsar?
> > > > Kiryl,
> > > >
> > > > Il giorno lun 20 feb 2023 alle ore 12:18 Kiryl Valkovich
> > > >  ha scritto:
> > > > >
> > > > > Enrico, it seems you read only the mail message title.
> > > >
> > > > Sorry about that,
> > > > I have re-read the message, and I have just realised that I skipped
> > > > the very first line :-)
> > > >
> > > > I have one question.
> > > > IIUC you started from scratch a new UI, could you please explain why
> > > > Pulsar Manager doesn't work for you?
> > > >
> > > >
> > > > Cheers
> > > > Enrico
> > > >
> > > > >
> > > > > From: Enrico Olivelli 
> > > > > Date: Monday, February 20, 2023 at 11:59 AM
> > > > > To: dev@pulsar.apache.org 
> > > > > Subject: Re: Does anyone build UI for Pulsar?
> > > > > Kiryl,
> > > > >
> > > > > You can use the official Apache Pulsar Manager the is maintained by
> > > > > this community
> > > > > https://github.com/apache/pulsar-manager
> > > > >
> > > > > At DataStax we also maintain this other UI that is also 100%
> > opensource
> > > > > https://github.com/datastax/pulsar-admin-console
> > > > >
> > > > > For the BookKeeper part there is BKVM (BookKeeper Visual Manager)
> > > > > https://github.com/diennea/bookkeeper-visual-manager
> > > > > This is also bundled with Apache Pulsar Manager
> > > > >
> > > > >
> > > > > I hope that helps
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > > Il 

Re: [Vote] PIP-245: Make subscriptions of non-persistent topic non-durable

[Jiuming Tao]

It makes sense

> 2023年2月20日 17:32，Zike Yang  写道：
> 
> Hi, Jiuming,
> 
> Overall looks good to me. Left some comments for the Compatibility section:
> 
>> In the next release after 2.11.1, if users want to create Durable 
>> subscriptions on NonPersistentTopic, will throw an exception.
> 
> I think this is an improvement but not a bug fix. Right?
> The next feature release is 3.0.0. I think we should print warn logs
> in 3.0.0 and throw exceptions may be in 3.1.0. Otherwise, it will
> bring the breaking change here in 3.0.0.
> The 2.11.1 is a patch release. I don't recommend cherry-picking this
> PIP to branch-2.11 because it's not a critical bug fix.
> 
> BR,
> Zike Yang
> 
> 
> Zike Yang
> 
> On Fri, Feb 17, 2023 at 12:52 AM Baodi Shi  wrote:
>> 
>> +1 (non-binding)
>> 
>> Thanks,
>> Baodi Shi
>> 
>> 
>> 在 2023年2月16日 21:44:17 上，Asaf Mesika  写道：
>> 
>>> +1 (non-binding)
>>> 
>>> 
>>> On Thu, Feb 16, 2023 at 11:11 AM Jiuming Tao >>> 
>>> wrote:
>>> 
>>> 
>>> I’ve added the `Compatibility` selection into the PIP, please help review
>>> 
>>> and vote the PIP
>>> 
>>> 
>>> Thanks,
>>> 
>>> Tao Jiuming
>>> 
>>> 
>>> 
>>> 
 2023年2月15日 14:58，Zike Yang  写道：
>>> 
 
>>> 
 Hi, Jiuming
>>> 
 
>>> 
> bump
>>> 
 
>>> 
 As for the discussion here[0], could you add a `Compatibility` section
>>> 
 to talk about compatibility in more detail? WDYT?
>>> 
 Then we could start the vote again.
>>> 
 
>>> 
 [0] https://lists.apache.org/thread/2bjg39zh7z38bzbnqngbo5l4jzkjttrq
>>> 
 
>>> 
 Thanks,
>>> 
 Zike Yang
>>> 
 
>>> 
 On Wed, Feb 15, 2023 at 1:34 PM Tao Jiuming  wrote:
>>> 
> 
>>> 
> 
>>> 
> bump
>>> 
> 
>>> 
> On 2023/02/13 06:56:09 Jiuming Tao wrote:
>>> 
>> Hi all,
>>> 
>> 
>>> 
>> I would like to start a VOTE on `PIP-245: Make subscriptions of
>>> 
>>> non-persistent topic non-durable`.
>>> 
>> 
>>> 
>> Motivation:
>>> 
>> 
>>> 
>> There are two types of subscriptions for a topic: Durable and
>>> 
>>> Non-durable.
>>> 
>> 
>>> 
>> We create a Consumer with a Durable subscription and a Reader with a
>>> 
>>> Non-durable subscription.
>>> 
>> 
>>> 
>> But for NonPersistentTopic, creating a Durable subscription is
>>> 
>>> meaningless, NonPersistentSubscription doesn't have a ManagedCursor to
>>> 
>>> persistent its data. After its consumer disconnected, the subscription
>>> 
>>> couldn't be removed automatically if we didn't set the value of
>>> 
>>> subscriptionExpirationTimeMinutes greater than 0.
>>> 
>> 
>>> 
>> For subscriptionExpirationTimeMinutes, it controls the subscription
>>> 
>>> expiration of NonPersistentTopic and PersistentTopic, if we set the value
>>> 
>>> of subscriptionExpirationTimeMinutes greater than 0, it may lead to data
>>> 
>>> loss(The durable subscriptions of PersistentTopic also can be removed).
>>> 
>> 
>>> 
>> And the Non-durable subscriptions will be removed automatically after
>>> 
>>> all the consumers disconnected, it's the existing logic.
>>> 
>> 
>>> 
>> For the purpose of removing the subscriptions which have no active
>>> 
>>> consumers of NonPersistentTopic and the above reasons, we can make all the
>>> 
>>> subscriptions of a NonPersistentTopic Non-durable.
>>> 
>> 
>>> 
>> 
>>> 
>> 
>>> 
>> For more details, you can read:
>>> 
>>> https://github.com/apache/pulsar/issues/19448 <
>>> 
>>> https://github.com/apache/pulsar/issues/19448>
>>> 
>> 
>>> 
>> And the discuss thread is available at:
>>> 
>>> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst <
>>> 
>>> https://lists.apache.org/thread/2ltmyglnb25jy8nk58twkwbglws43bst>
>>> 
>> 
>>> 
>> Thanks,
>>> 
>> Tao Jiuming
>>> 
>>> 
>>> 
>>> 

Re: [DISCUSS] Using bouncycastle fips instead bouncycastle non-fips

[Zixuan Liu]

> 1. What is FIPS?

FIPS (Federal Information Processing Standards) are a set of standards that
describe document processing, encryption algorithms and other information
technology standards for use within non-military government agencies and by
government contractors and vendors who work with the agencies.

> 2. Why is the FIPS version safer exactly?

FIPS standard is strict. When using the FIPS version, this is also very
strict and standard.

> 3. What is bouncycastle used exactly in Pulsar?

We use the bouncycastle as the TLS provider,  and used for the end-to-end
message encryption.

Thanks,
Zixuan

Asaf Mesika  于2023年2月22日周三 21:23写道：

> Can you elaborate a bit:
> 1. What is FIPS?
> 2. Why is the FIPS version safer exactly?
> 3. What is bouncycastle used exactly in Pulsar?
>
>
>
> On Wed, Feb 22, 2023 at 11:58 AM Zixuan Liu  wrote:
>
> > Hi all,
> >
> > I would like to discuss using the bouncycastle fips instead of the
> > bouncycastle non-fips.
> >
> > The bouncycastle is a Java library that complements the default Java
> > Cryptographic Extension (JCE), which has two versions: fips version and
> > non-fips version.
> >
> > The fips version is safer than non-fips. When the security level is very
> > high, many policies require the fips version, but the Pulsar default uses
> > the non-fips version. Switch this is complex, because
> > the `pulsar-client-messagecrypto-bc` module and root project depends on
> the
> > non-fips, so I suggest we switch to fips version from non-fips.
> >
> > Reference:
> > - https://www.bouncycastle.org/
> > - https://www.bouncycastle.org/fips_faq.html
> > - https://en.wikipedia.org/wiki/Federal_Information_Processing_Standards
> >
> > Thanks,
> > Zixuan
> >
>

Re: [DISCUSS] PIP-247: Notifications for partitions update

[Asaf Mesika]

How about edge cases?
In Andra's PIP he took into account cases where updates were lost, so he
created a secondary poll. Not saying it's the best situation for your case
of course.
I'm saying that when a broker sends an update CommandPartitionUpdate, how
do you know it arrived successfully? From my memory, there is no ACK in the
protocol, saying "I'm the client, I got the update successfully" and only
then it removed the "dirty" flag for that topic, for this watcher ID.

Are there any other edge cases we can have? Let's be exhaustive.



On Wed, Feb 22, 2023 at 1:14 PM houxiaoyu  wrote:

> Thanks for your great suggestion Enrico.
>
> I agreed with you. It's more reasonable to add a
> `supports_partition_update_watchers`  in `FeatureFlags`  to detect that the
> connected broker supporting this feature , and add a new broker
> configuration property `enableNotificationForPartitionUpdate` with default
> value true, which is much like PIP-145.
>
> I have updated the descriptions.
>
> Enrico Olivelli  于2023年2月22日周三 17:26写道：
>
> > I support this proposal.
> > Coping here my comments from GH:
> >
> > can't we enable this by default in case we detect that the connected
> > Broker supports it ?
> > I can't find any reason for not using this mechanism if it is available.
> >
> > Maybe we can set the default to "true" and allow users to disable it
> > in case it impacts their systems in an unwanted way.
> >
> > Maybe It would be useful to have a way to disable the mechanism on the
> > broker side as well
> >
> > Enrico
> >
> > Il giorno mer 22 feb 2023 alle ore 10:22 houxiaoyu
> >  ha scritto:
> > >
> > > Hi Pulsar community:
> > >
> > > I opened a PIP to discuss "Notifications for partitions update"
> > >
> > > ### Motivation
> > >
> > > Pulsar client will poll brokers at fix time for checking the partitions
> > > update if we publish/subscribe the partitioned topics with
> > > `autoUpdatePartitions` as true. This causes unnecessary load for  both
> > > clients and brokers since most of the time the number of partitions
> will
> > > not change. In addition polling introduces latency in partitions update
> > >  which is specified by `autoUpdatePartitionsInterval`.
> > > This PIP would like to introduce a notification mechanism for partition
> > > update, which is much like PIP-145 for regex subscriptions
> > > https://github.com/apache/pulsar/issues/14505.
> > >
> > > For more details, please read the PIP at:
> > > https://github.com/apache/pulsar/issues/19596
> > > Looking forward to hearing your thoughts.
> > >
> > > Thanks,
> > > Xiaoyu Hou
> > > 
> >
>

Re: [DISCUSS] Using bouncycastle fips instead bouncycastle non-fips

[Asaf Mesika]

Can you elaborate a bit:
1. What is FIPS?
2. Why is the FIPS version safer exactly?
3. What is bouncycastle used exactly in Pulsar?



On Wed, Feb 22, 2023 at 11:58 AM Zixuan Liu  wrote:

> Hi all,
>
> I would like to discuss using the bouncycastle fips instead of the
> bouncycastle non-fips.
>
> The bouncycastle is a Java library that complements the default Java
> Cryptographic Extension (JCE), which has two versions: fips version and
> non-fips version.
>
> The fips version is safer than non-fips. When the security level is very
> high, many policies require the fips version, but the Pulsar default uses
> the non-fips version. Switch this is complex, because
> the `pulsar-client-messagecrypto-bc` module and root project depends on the
> non-fips, so I suggest we switch to fips version from non-fips.
>
> Reference:
> - https://www.bouncycastle.org/
> - https://www.bouncycastle.org/fips_faq.html
> - https://en.wikipedia.org/wiki/Federal_Information_Processing_Standards
>
> Thanks,
> Zixuan
>

Re: Does anyone build UI for Pulsar?

[Asaf Mesika]

Wondering out loud if there are generic UI for message brokers?

On Tue, Feb 21, 2023 at 6:27 PM Kiryl Valkovich 
wrote:

> Devin,
>
> The most desired feature I need as an event-driven application developer
> is the ability to read and filter messages in topics.
> With the NonDurable subscription type, it shouldn’t affect the topic’s
> data.
>
> The ability to easily produce a message or generate a bunch of them, based
> on some rules, allows me to quickly check how the application behaves in
> different scenarios.
>
> It’s also useful for the QA audience.
>
> From: Devin Bost 
> Date: Tuesday, February 21, 2023 at 4:58 PM
> To: Dev 
> Subject: Re: Does anyone build UI for Pulsar?
> Kiryl,
>
> > Just make a side-by-side comparison of Pulsar Manager with any of the
> following options:
> > - Conduktor (Commercial).
> > - Kafka UI by Provectus (Apache License 2.0).
> > - Redpanda Console (ex-Kowl) (Mixed license).
> >
> > I see a significant gap in user experience between any of them and the
> Pulsar Manager.
>
> Thanks for bringing up this subject. You're making good points that we have
> a gap in this area.
>
> In your perspective, where do you think the biggest gaps are in terms of
> functionality?
>
> --
> Devin Bost
> Sent from mobile
> Cell: 801-400-4602
>
> On Mon, Feb 20, 2023, 8:54 AM Enrico Olivelli  wrote:
>
> > Il giorno lun 20 feb 2023 alle ore 14:41 Kiryl Valkovich
> >  ha scritto:
> > >
> > > Enrico, it’s easy.
> > >
> > > When I tried it, the basic functionality just didn’t work.
> > >
> > > Just make a side-by-side comparison of Pulsar Manager with any of the
> > following options:
> > > - Conduktor (Commercial).
> > > - Kafka UI by Provectus (Apache License 2.0).
> > > - Redpanda Console (ex-Kowl) (Mixed license).
> > >
> > > I see a significant gap in user experience between any of them and the
> > Pulsar Manager.
> > > Also, I just don’t see much contribution activity.
> > >
> > > Redpanda Console and Kafka UI, each have about x10 times more commits
> in
> > a year, than Pulsar Manager has in the last 3 years. Therefore I can’t
> > conclude that the project is alive.
> >
> > (Unfortunately) I 100% agree with you.
> >
> > >
> > > Regarding the “Why from scratch?” question:
> > > - I found it easier for me personally because of the different tech
> > stack I use.
> > > - Starting from scratch is a good opportunity to learn Pulsar itself.
> > > - I’d want to make a product around that.
> > >
> > > I'm not claiming that my result will be 100% better, but why not try?
> >
> > I understand your points.
> > I hope you will share your project with OSS license, this way it will
> > be easier for Pulsar users to try it out and contribute.
> > >
> > > If someone is already doing or plans to do similar work, let’s
> > collaborate!
> >
> > Thanks for sharing
> >
> > Enrico
> >
> > >
> > > From: Enrico Olivelli 
> > > Date: Monday, February 20, 2023 at 2:04 PM
> > > To: dev@pulsar.apache.org 
> > > Subject: Re: Does anyone build UI for Pulsar?
> > > Kiryl,
> > >
> > > Il giorno lun 20 feb 2023 alle ore 12:18 Kiryl Valkovich
> > >  ha scritto:
> > > >
> > > > Enrico, it seems you read only the mail message title.
> > >
> > > Sorry about that,
> > > I have re-read the message, and I have just realised that I skipped
> > > the very first line :-)
> > >
> > > I have one question.
> > > IIUC you started from scratch a new UI, could you please explain why
> > > Pulsar Manager doesn't work for you?
> > >
> > >
> > > Cheers
> > > Enrico
> > >
> > > >
> > > > From: Enrico Olivelli 
> > > > Date: Monday, February 20, 2023 at 11:59 AM
> > > > To: dev@pulsar.apache.org 
> > > > Subject: Re: Does anyone build UI for Pulsar?
> > > > Kiryl,
> > > >
> > > > You can use the official Apache Pulsar Manager the is maintained by
> > > > this community
> > > > https://github.com/apache/pulsar-manager
> > > >
> > > > At DataStax we also maintain this other UI that is also 100%
> opensource
> > > > https://github.com/datastax/pulsar-admin-console
> > > >
> > > > For the BookKeeper part there is BKVM (BookKeeper Visual Manager)
> > > > https://github.com/diennea/bookkeeper-visual-manager
> > > > This is also bundled with Apache Pulsar Manager
> > > >
> > > >
> > > > I hope that helps
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > > Il giorno lun 20 feb 2023 alle ore 10:51 Kiryl Valkovich
> > > >  ha scritto:
> > > > >
> > > > > Hi everyone!
> > > > >
> > > > > Does anyone personally or some company work on UI for Pulsar other
> > than pulsar-manager or pulsar-admin-console?
> > > > >
> > > > > I understand that StreamNative and DataStax have managed solutions
> > and obviously work on their UI.
> > > > >
> > > > > I rather looking for an open-source or commercial tool that can be
> > used in pair with any Pulsar deployment.
> > > > >
> > > > > I spent some time implementing UI for Apache Pulsar. It’s not ready
> > to release yet. As usual, the most difficult 20% of the work remained.
> > > > >
> > > > > I’m asking that 

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

[Asaf Mesika]

I read it but you're almost directly diving into the code - it will take me
hours just to reverse engineer your design.

Can you please include a "High Level Design" section in which you explain
how you plan to tackle any issue?
If I can read that section and explain to someone else how this will work,
it means the section is complete.

Let's leave the code to the PRs.


On Sun, Feb 19, 2023 at 2:59 PM SiNan Liu  wrote:

> Hi all,
>
> I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565.
>
> We can talk about the current design here. Especially for the field type
> change check rules, please give your valuable advice.
>
> Thanks,
> Sinan
>

Re: Force redirect questions from Slack to GitHub Discussions or StackOverflow?

[Asaf Mesika]

No Kiryl, I meant the bot will message the user saying - "This message (It
opens a thread) looks like a question. You can use the bot to post this
question to GitHub for you. Just click the three dots, and choose `Post to
GitHub Discussion`"


On Mon, Feb 20, 2023 at 5:58 PM Dave Fisher  wrote:

> We do not have the right to move an individual’s message to a platform
> where they have not agreed to the terms of use. We must not “force
> redirect”. We don’t own these questions, the person asking the question
> does.
>
> +1 to pinning a message to Slack channels.
>
> Best,
> Dave
>
> Sent from my iPhone
>
> > On Feb 20, 2023, at 2:43 AM, Kiryl Valkovich 
> wrote:
> >
> > Do you mean, to do it for all messages in the #general channel (maybe
> only for messages that contain the question mark)?
> >
> > I think it makes sense.
> >
> > From: Asaf Mesika 
> > Date: Sunday, February 19, 2023 at 11:54 AM
> > To: dev@pulsar.apache.org 
> > Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or StackOverflow?
> > I would have the bot open a Thread for the message, *suggesting* the user
> > to click to convert this question into a GitHub Discussion question. This
> > way you can have the actual GitHub user asking the question and not a bot
> > one.
> >
> >> On Fri, Feb 17, 2023 at 10:59 PM Kiryl Valkovich
> 
> >> wrote:
> >>
> >> What about such wording?
> >>
> >> ---
> >> Your question was moved here:
> >> https://github.com/apache/pulsar/discussions/123
> >>
> >> Please consider asking new questions here:
> >>
> >>  *   At StackOverflow using apache-pulsar tag.
> >>  *   In the Q category at GitHub Discussions.
> >>  *   Apache Pulsar User Mailing List.
> >>
> >>
> >> It will make it searchable by others. Also, this way we can collect a
> >> knowledge base outside of Slack over time.
> >>
> >> I can’t see how the words “please consider” force the user to do
> something.
> >>
> >> Users who have an account on StackOverflow or GitHub can use these
> >> platforms next time.
> >> Others can send their question via the mailing list.
> >>
> >> From: Dave Fisher 
> >> Date: Friday, February 17, 2023 at 9:28 PM
> >> To: dev@pulsar.apache.org 
> >> Subject: Re: Force redirect questions from Slack to GitHub Discussions
> or
> >> StackOverflow?
> >> My concern is that users should have a choice on where to post their
> >> questions. They might have concerns about GitHub’s terms and
> conditions. We
> >> can pin a message to slack pointing to GitHub discussions and
> stackoverflow.
> >>
> >> Best,
> >> Dave
> >>
> >> Sent from my iPhone
> >>
> >>> On Feb 17, 2023, at 9:22 AM, Kiryl Valkovich
> 
> >> wrote:
> >>>
> >>> I’m the owner of this account.
> >>> The goal is to test drive duplicating Slack questions to the GitHub
> >> discussions.
> >>> With the current level of activity in Slack it’s not so hard to do it
> >> manually.
> >>>
> >>> I’m in CET now. I can share the account credentials with people who can
> >> post questions to GitHub Discussions on behalf of this account in other
> >> time zones.
> >>> Or I can do it once a day.
> >>>
> >>> If someone doesn’t find it useful or has ideas on how to do it in a
> >> better way, just say it directly.
> >>>
> >>> From: Enrico Olivelli 
> >>> Date: Friday, February 17, 2023 at 3:43 PM
> >>> To: dev@pulsar.apache.org 
> >>> Subject: Re: Force redirect questions from Slack to GitHub Discussions
> >> or StackOverflow?
> >>> Hello,
> >>> I see that some "Pulsar Community Bot" appeared in Slack
> >>>
> >>> it is connected to this email address "pulsar.community@gmail.com"
> >>>
> >>> While I find this thing "amazing"I wonder if I missed something,
> >>> who is the owner of this "bot" ?
> >>>
> >>>
> >>> Enrico
> >>>
>  Il giorno gio 16 feb 2023 alle ore 16:03 Kiryl Valkovich
>   ha scritto:
> 
>  Played with Slack and StackOverflow APIs a bit.
> 
>  The Slack API works as expected. After clicking the message button, it
> >> sends a message descriptor to my app where I can do anything with its
> >> content.
> 
>  It’s possible to post messages via the StackOverflow API, but it’s
> >> unlikely that any Slack message can be converted to a StackOverflow
> >> question.
> 
>  I encountered several types of validation errors:
> 
>  -  This question body does not meet our quality standards.
> >> Please make sure that it completely describes your problem - including
> what
> >> you have already tried - and is written using proper grammar.
> 
>  *   Something similar to “This message looks like a duplicate of
> >> another message”.
> 
>  I believe, GitHub Discussions don’t have such kind of “quality
> >> standards” validation.
> 
>  From: Kiryl Valkovich 
>  Date: Thursday, February 16, 2023 at 1:33 PM
>  To: dev@pulsar.apache.org 
>  Subject: Re: Force redirect questions from Slack to GitHub Discussions
> >> or StackOverflow?
>  If there are no hidden obstacles, we 

Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.

[Asaf Mesika]

How are those stats exposed to the user?


On Mon, Feb 20, 2023 at 6:01 AM Xiangying Meng  wrote:

> Hi, Community,
> We plan to add an internal class to `TransactionBufferStats` to record the
> snapshot status uniformly.
> As we all know, the current transaction buffer(TB) filters the messages
> sent using the aborted transaction by storing the aborted ID in TB.
> Then TB will periodically store these aborted txn IDs in a bookie entry in
> the form of snapshots so that TB can recover faster when recovering.
> But as more and more people use transactions, we found that in some extreme
> cases, a bookie entry may not be able to store all aborted transaction IDs.
> So in PIP196 , we
> implemented the multiple-snapshot function.
> As the transaction buffer snapshot mechanism becomes increasingly complex,
> the only information related to the transaction snapshot is
> `lastSnapshotTimestamps`; That is not enough, we need to add more info to
> record the snapshot stats.
> So I suggest adding an internal class SnapshotStats to
> TransactionBufferStats to record the snapshot status uniformly.
>
> The modification could be :
> ```java
> public class TransactionBufferStats {
> ...
> public long lastSnapshotTimestamps;
> ...
> }
> ```
> ```java
> public class TransactionBufferStats {
> ...
> //public long lastSnapshotTimestamps;
> ...
> public SnapshotStats snapshotStats;
>
> public static class SnapshotStats {
> public long segmentsSize;
>
> public long unsealedAbortTxnIDs;
>
>
> public long lastSnapshotTimestamps;
> }
> }
>
> ```
> Thanks.
> Xiangying
>

Re: [DISCUSS] PIP-247: Notifications for partitions update

[houxiaoyu]

Thanks for your great suggestion Enrico.

I agreed with you. It's more reasonable to add a
`supports_partition_update_watchers`  in `FeatureFlags`  to detect that the
connected broker supporting this feature , and add a new broker
configuration property `enableNotificationForPartitionUpdate` with default
value true, which is much like PIP-145.

I have updated the descriptions.

Enrico Olivelli  于2023年2月22日周三 17:26写道：

> I support this proposal.
> Coping here my comments from GH:
>
> can't we enable this by default in case we detect that the connected
> Broker supports it ?
> I can't find any reason for not using this mechanism if it is available.
>
> Maybe we can set the default to "true" and allow users to disable it
> in case it impacts their systems in an unwanted way.
>
> Maybe It would be useful to have a way to disable the mechanism on the
> broker side as well
>
> Enrico
>
> Il giorno mer 22 feb 2023 alle ore 10:22 houxiaoyu
>  ha scritto:
> >
> > Hi Pulsar community:
> >
> > I opened a PIP to discuss "Notifications for partitions update"
> >
> > ### Motivation
> >
> > Pulsar client will poll brokers at fix time for checking the partitions
> > update if we publish/subscribe the partitioned topics with
> > `autoUpdatePartitions` as true. This causes unnecessary load for  both
> > clients and brokers since most of the time the number of partitions will
> > not change. In addition polling introduces latency in partitions update
> >  which is specified by `autoUpdatePartitionsInterval`.
> > This PIP would like to introduce a notification mechanism for partition
> > update, which is much like PIP-145 for regex subscriptions
> > https://github.com/apache/pulsar/issues/14505.
> >
> > For more details, please read the PIP at:
> > https://github.com/apache/pulsar/issues/19596
> > Looking forward to hearing your thoughts.
> >
> > Thanks,
> > Xiaoyu Hou
> > 
>

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

[Enrico Olivelli]

The problem reported by Nicolò seems a real show stopper
-1 (binding)

Is there any volunteer on fixing the problem ?

I think that most of Production users use TLS and this problem prevent
anyone to upgrade from 1.7.x to 1.8.x

Enrico

Il giorno mer 22 feb 2023 alle ore 10:33 Nicolò Boschi
 ha scritto:
>
> Yes, I'm on MacOS
>
> Adding the explicit path to the system CA worked, thanks.
>
> I wonder why it wasn't needed in 1.7.0 and now it's required in 1.8.1.
> This can be problematic for users because the client upgrade requires code
> changes and redeployment.
> Maybe it's a regression from 1.8.0, but I think that we need to get that
> fixed in 1.8.1
>
> Nicolò Boschi
>
>
> Il giorno mer 22 feb 2023 alle ore 10:17 Baodi Shi  ha
> scritto:
>
> > Hi, @Nicolò Boschi 
> >
> > Is your system macOS(arm64)?
> >
> > Maybe you should set tlsTrustCertsFilePath:
> >
> >   const client = new Pulsar.Client({
> > tlsTrustCertsFilePath:'/etc/ssl/cert.pem',
> >   });
> >
> >
> >
> >
> > Thanks,
> > Baodi Shi
> >
> >
> > 在 2023年2月22日 15:59:28 上，Nicolò Boschi  写道：
> >
> >> Hi,
> >>
> >> I'm having issues while validating the fix related to the hostname
> >> verification: https://github.com/apache/pulsar-client-cpp/pull/126
> >> My usecase is with a valid TLS certificate signed by a CA (not a
> >> self-signed one).
> >>
> >> My code is very simple (see below): it creates a client with token auth +
> >> TLS and sends some messages.
> >>
> >> It works well with node client 1.7.0 with cpp client 3.1.2
> >> It fails with node client 1.8.0 (as expected)
> >> It still fails with the rc: 1.8.1-rc.1
> >>
> >> (I'm installing the dependency with "npm i pulsar-client@1.8.1-rc.1
> >> --pulsar_binary_host_mirror=
> >> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/;
> >>
> >> The error I'm seeing is this one:
> >>
> >> [INFO][ClientConnection:388] Connected to broker
> >> [ERROR][ClientConnection:488] Handshake failed: certificate verify failed
> >> (SSL routines, tls_process_server_certificate)
> >> [INFO][ClientConnection:1600] Connection closed with ConnectError
> >>
> >> Note that setting `tlsValidateHostname: true` "resolves" the problem,
> >> however it's not acceptable as you know.
> >>
> >> I'm pretty sure that it's related to the cpp client dependency, however
> >> I'm
> >> not very familiar with it and how it's bundled in the node client >= 1.8.0
> >> Is there a way to verify if the bundled cpp client is actually the
> >> expected one?
> >>
> >>
> >>
> >> This is the code snippet:
> >> ```
> >> const tokenStr = asToken;
> >>  const pulsarUri = pulsarUrl;
> >>  const topicName = asTopic;
> >>
> >>  const auth = new Pulsar.AuthenticationToken({ token: tokenStr });
> >>  const client = new Pulsar.Client({
> >>serviceUrl: pulsarUri,
> >>authentication: auth,
> >>operationTimeoutSeconds: 30,
> >>tlsCertificateFilePath: "",
> >>tlsValidateHostname: false
> >>  });
> >>  Pulsar.Client.setLogHandler((level, file, line, message) => {
> >>console.log('[%s][%s:%d] %s', Pulsar.LogLevel.toString(level), file,
> >> line, message);
> >>  });
> >>
> >>  const producer = await client.createProducer({
> >>topic: topicName,
> >>  })
> >>
> >>  for (let i = 0; i < 10; i += 1) {
> >>await producer.send({
> >>  data: Buffer.from("nodejs-message-" + i),
> >>});
> >>console.log("send message " + i);
> >>  }
> >>  await producer.flush();
> >>  await producer.close();
> >>  await client.close();
> >>
> >> ```
> >>
> >> Thanks,
> >> Nicolò Boschi
> >>
> >>
> >> Il giorno mer 22 feb 2023 alle ore 08:02 Yunze Xu
> >>  ha scritto:
> >>
> >> +1 (binding)
> >>
> >> * Verified checksum and signature
> >>
> >> * Build from source
> >>
> >> * Install from npm on Ubuntu 20.04
> >>
> >> * Run an end-to-end test with custom `tlsTrustCertsFilePath` config on
> >>
> >> StreamNative cloud with OAuth2 authentication
> >>
> >>
> >> BTW, from the discussion here [1], it would be better to use
> >>
> >> https://downloads.apache.org/pulsar/KEYS as the KEYS,
> >>
> >>
> >> [1] https://lists.apache.org/thread/f9w430oqpm0g72b1htwbtc8y3mfqf8r6
> >>
> >>
> >> Thanks,
> >>
> >> Yunze
> >>
> >>
> >> On Mon, Feb 20, 2023 at 5:36 PM Nozomi Kurihara 
> >>
> >> wrote:
> >>
> >> >
> >>
> >> > +1 (binding)
> >>
> >> >
> >>
> >> > * checked license headers
> >>
> >> > * verified checksum and signature
> >>
> >> > * install from npm and run producer/consumer
> >>
> >> >
> >>
> >> > Thanks,
> >>
> >> > Nozomi
> >>
> >> >
> >>
> >> > 2023年2月17日(金) 19:12 Baodi Shi :
> >>
> >> >
> >>
> >> > > Hi everyone,
> >>
> >> > >
> >>
> >> > > This is the first release candidate for Apache Pulsar Node.js client,
> >>
> >> > > version 1.8.1.
> >>
> >> > >
> >>
> >> > > It fixes the following
> >>
> >> > > issues:
> >>
> >> > >
> >>
> >>
> >> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.1+is%3Aclosed
> >>
> >> > >
> >>
> >> > > Please download the source files and review this release candidate:
> 

[DISCUSS] Using bouncycastle fips instead bouncycastle non-fips

[Zixuan Liu]

Hi all,

I would like to discuss using the bouncycastle fips instead of the
bouncycastle non-fips.

The bouncycastle is a Java library that complements the default Java
Cryptographic Extension (JCE), which has two versions: fips version and
non-fips version.

The fips version is safer than non-fips. When the security level is very
high, many policies require the fips version, but the Pulsar default uses
the non-fips version. Switch this is complex, because
the `pulsar-client-messagecrypto-bc` module and root project depends on the
non-fips, so I suggest we switch to fips version from non-fips.

Reference:
- https://www.bouncycastle.org/
- https://www.bouncycastle.org/fips_faq.html
- https://en.wikipedia.org/wiki/Federal_Information_Processing_Standards

Thanks,
Zixuan

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

[Nicolò Boschi]

Yes, I'm on MacOS

Adding the explicit path to the system CA worked, thanks.

I wonder why it wasn't needed in 1.7.0 and now it's required in 1.8.1.
This can be problematic for users because the client upgrade requires code
changes and redeployment.
Maybe it's a regression from 1.8.0, but I think that we need to get that
fixed in 1.8.1

Nicolò Boschi


Il giorno mer 22 feb 2023 alle ore 10:17 Baodi Shi  ha
scritto:

> Hi, @Nicolò Boschi 
>
> Is your system macOS(arm64)?
>
> Maybe you should set tlsTrustCertsFilePath:
>
>   const client = new Pulsar.Client({
> tlsTrustCertsFilePath:'/etc/ssl/cert.pem',
>   });
>
>
>
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年2月22日 15:59:28 上，Nicolò Boschi  写道：
>
>> Hi,
>>
>> I'm having issues while validating the fix related to the hostname
>> verification: https://github.com/apache/pulsar-client-cpp/pull/126
>> My usecase is with a valid TLS certificate signed by a CA (not a
>> self-signed one).
>>
>> My code is very simple (see below): it creates a client with token auth +
>> TLS and sends some messages.
>>
>> It works well with node client 1.7.0 with cpp client 3.1.2
>> It fails with node client 1.8.0 (as expected)
>> It still fails with the rc: 1.8.1-rc.1
>>
>> (I'm installing the dependency with "npm i pulsar-client@1.8.1-rc.1
>> --pulsar_binary_host_mirror=
>> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/;
>>
>> The error I'm seeing is this one:
>>
>> [INFO][ClientConnection:388] Connected to broker
>> [ERROR][ClientConnection:488] Handshake failed: certificate verify failed
>> (SSL routines, tls_process_server_certificate)
>> [INFO][ClientConnection:1600] Connection closed with ConnectError
>>
>> Note that setting `tlsValidateHostname: true` "resolves" the problem,
>> however it's not acceptable as you know.
>>
>> I'm pretty sure that it's related to the cpp client dependency, however
>> I'm
>> not very familiar with it and how it's bundled in the node client >= 1.8.0
>> Is there a way to verify if the bundled cpp client is actually the
>> expected one?
>>
>>
>>
>> This is the code snippet:
>> ```
>> const tokenStr = asToken;
>>  const pulsarUri = pulsarUrl;
>>  const topicName = asTopic;
>>
>>  const auth = new Pulsar.AuthenticationToken({ token: tokenStr });
>>  const client = new Pulsar.Client({
>>serviceUrl: pulsarUri,
>>authentication: auth,
>>operationTimeoutSeconds: 30,
>>tlsCertificateFilePath: "",
>>tlsValidateHostname: false
>>  });
>>  Pulsar.Client.setLogHandler((level, file, line, message) => {
>>console.log('[%s][%s:%d] %s', Pulsar.LogLevel.toString(level), file,
>> line, message);
>>  });
>>
>>  const producer = await client.createProducer({
>>topic: topicName,
>>  })
>>
>>  for (let i = 0; i < 10; i += 1) {
>>await producer.send({
>>  data: Buffer.from("nodejs-message-" + i),
>>});
>>console.log("send message " + i);
>>  }
>>  await producer.flush();
>>  await producer.close();
>>  await client.close();
>>
>> ```
>>
>> Thanks,
>> Nicolò Boschi
>>
>>
>> Il giorno mer 22 feb 2023 alle ore 08:02 Yunze Xu
>>  ha scritto:
>>
>> +1 (binding)
>>
>> * Verified checksum and signature
>>
>> * Build from source
>>
>> * Install from npm on Ubuntu 20.04
>>
>> * Run an end-to-end test with custom `tlsTrustCertsFilePath` config on
>>
>> StreamNative cloud with OAuth2 authentication
>>
>>
>> BTW, from the discussion here [1], it would be better to use
>>
>> https://downloads.apache.org/pulsar/KEYS as the KEYS,
>>
>>
>> [1] https://lists.apache.org/thread/f9w430oqpm0g72b1htwbtc8y3mfqf8r6
>>
>>
>> Thanks,
>>
>> Yunze
>>
>>
>> On Mon, Feb 20, 2023 at 5:36 PM Nozomi Kurihara 
>>
>> wrote:
>>
>> >
>>
>> > +1 (binding)
>>
>> >
>>
>> > * checked license headers
>>
>> > * verified checksum and signature
>>
>> > * install from npm and run producer/consumer
>>
>> >
>>
>> > Thanks,
>>
>> > Nozomi
>>
>> >
>>
>> > 2023年2月17日(金) 19:12 Baodi Shi :
>>
>> >
>>
>> > > Hi everyone,
>>
>> > >
>>
>> > > This is the first release candidate for Apache Pulsar Node.js client,
>>
>> > > version 1.8.1.
>>
>> > >
>>
>> > > It fixes the following
>>
>> > > issues:
>>
>> > >
>>
>>
>> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.1+is%3Aclosed
>>
>> > >
>>
>> > > Please download the source files and review this release candidate:
>>
>> > > - Download the source package, verify shasum and asc
>>
>> > > - Follow the README.md to build and run the Pulsar Node.js client.
>>
>> > >
>>
>> > > The release candidate package has been published to the npm
>>
>> > > registry:https://www.npmjs.com/package/pulsar-client/v/1.8.1-rc.1
>>
>> > > You can install it by `npm i pulsar-client@1.8.1-rc.1
>>
>> > > --pulsar_binary_host_mirror=
>>
>> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>> 
>>
>> 
>>
>> > > 

Re: [DISCUSS] PIP-247: Notifications for partitions update

[Enrico Olivelli]

I support this proposal.
Coping here my comments from GH:

can't we enable this by default in case we detect that the connected
Broker supports it ?
I can't find any reason for not using this mechanism if it is available.

Maybe we can set the default to "true" and allow users to disable it
in case it impacts their systems in an unwanted way.

Maybe It would be useful to have a way to disable the mechanism on the
broker side as well

Enrico

Il giorno mer 22 feb 2023 alle ore 10:22 houxiaoyu
 ha scritto:
>
> Hi Pulsar community:
>
> I opened a PIP to discuss "Notifications for partitions update"
>
> ### Motivation
>
> Pulsar client will poll brokers at fix time for checking the partitions
> update if we publish/subscribe the partitioned topics with
> `autoUpdatePartitions` as true. This causes unnecessary load for  both
> clients and brokers since most of the time the number of partitions will
> not change. In addition polling introduces latency in partitions update
>  which is specified by `autoUpdatePartitionsInterval`.
> This PIP would like to introduce a notification mechanism for partition
> update, which is much like PIP-145 for regex subscriptions
> https://github.com/apache/pulsar/issues/14505.
>
> For more details, please read the PIP at:
> https://github.com/apache/pulsar/issues/19596
> Looking forward to hearing your thoughts.
>
> Thanks,
> Xiaoyu Hou
> 

[DISCUSS] PIP-247: Notifications for partitions update

[houxiaoyu]

Hi Pulsar community:

I opened a PIP to discuss "Notifications for partitions update"

### Motivation

Pulsar client will poll brokers at fix time for checking the partitions
update if we publish/subscribe the partitioned topics with
`autoUpdatePartitions` as true. This causes unnecessary load for  both
clients and brokers since most of the time the number of partitions will
not change. In addition polling introduces latency in partitions update
 which is specified by `autoUpdatePartitionsInterval`.
This PIP would like to introduce a notification mechanism for partition
update, which is much like PIP-145 for regex subscriptions
https://github.com/apache/pulsar/issues/14505.

For more details, please read the PIP at:
https://github.com/apache/pulsar/issues/19596
Looking forward to hearing your thoughts.

Thanks,
Xiaoyu Hou

Re: [VOTE] Pulsar Node.js Client Release 1.8.1 Candidate 1

[Baodi Shi]

Hi, @Nicolò Boschi 

Is your system macOS(arm64)?

Maybe you should set tlsTrustCertsFilePath:

  const client = new Pulsar.Client({
tlsTrustCertsFilePath:'/etc/ssl/cert.pem',
  });




Thanks,
Baodi Shi


在 2023年2月22日 15:59:28 上，Nicolò Boschi  写道：

> Hi,
>
> I'm having issues while validating the fix related to the hostname
> verification: https://github.com/apache/pulsar-client-cpp/pull/126
> My usecase is with a valid TLS certificate signed by a CA (not a
> self-signed one).
>
> My code is very simple (see below): it creates a client with token auth +
> TLS and sends some messages.
>
> It works well with node client 1.7.0 with cpp client 3.1.2
> It fails with node client 1.8.0 (as expected)
> It still fails with the rc: 1.8.1-rc.1
>
> (I'm installing the dependency with "npm i pulsar-client@1.8.1-rc.1
> --pulsar_binary_host_mirror=
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/;
>
> The error I'm seeing is this one:
>
> [INFO][ClientConnection:388] Connected to broker
> [ERROR][ClientConnection:488] Handshake failed: certificate verify failed
> (SSL routines, tls_process_server_certificate)
> [INFO][ClientConnection:1600] Connection closed with ConnectError
>
> Note that setting `tlsValidateHostname: true` "resolves" the problem,
> however it's not acceptable as you know.
>
> I'm pretty sure that it's related to the cpp client dependency, however I'm
> not very familiar with it and how it's bundled in the node client >= 1.8.0
> Is there a way to verify if the bundled cpp client is actually the
> expected one?
>
>
>
> This is the code snippet:
> ```
> const tokenStr = asToken;
>  const pulsarUri = pulsarUrl;
>  const topicName = asTopic;
>
>  const auth = new Pulsar.AuthenticationToken({ token: tokenStr });
>  const client = new Pulsar.Client({
>serviceUrl: pulsarUri,
>authentication: auth,
>operationTimeoutSeconds: 30,
>tlsCertificateFilePath: "",
>tlsValidateHostname: false
>  });
>  Pulsar.Client.setLogHandler((level, file, line, message) => {
>console.log('[%s][%s:%d] %s', Pulsar.LogLevel.toString(level), file,
> line, message);
>  });
>
>  const producer = await client.createProducer({
>topic: topicName,
>  })
>
>  for (let i = 0; i < 10; i += 1) {
>await producer.send({
>  data: Buffer.from("nodejs-message-" + i),
>});
>console.log("send message " + i);
>  }
>  await producer.flush();
>  await producer.close();
>  await client.close();
>
> ```
>
> Thanks,
> Nicolò Boschi
>
>
> Il giorno mer 22 feb 2023 alle ore 08:02 Yunze Xu
>  ha scritto:
>
> +1 (binding)
>
> * Verified checksum and signature
>
> * Build from source
>
> * Install from npm on Ubuntu 20.04
>
> * Run an end-to-end test with custom `tlsTrustCertsFilePath` config on
>
> StreamNative cloud with OAuth2 authentication
>
>
> BTW, from the discussion here [1], it would be better to use
>
> https://downloads.apache.org/pulsar/KEYS as the KEYS,
>
>
> [1] https://lists.apache.org/thread/f9w430oqpm0g72b1htwbtc8y3mfqf8r6
>
>
> Thanks,
>
> Yunze
>
>
> On Mon, Feb 20, 2023 at 5:36 PM Nozomi Kurihara 
>
> wrote:
>
> >
>
> > +1 (binding)
>
> >
>
> > * checked license headers
>
> > * verified checksum and signature
>
> > * install from npm and run producer/consumer
>
> >
>
> > Thanks,
>
> > Nozomi
>
> >
>
> > 2023年2月17日(金) 19:12 Baodi Shi :
>
> >
>
> > > Hi everyone,
>
> > >
>
> > > This is the first release candidate for Apache Pulsar Node.js client,
>
> > > version 1.8.1.
>
> > >
>
> > > It fixes the following
>
> > > issues:
>
> > >
>
>
> https://github.com/apache/pulsar-client-node/pulls?q=is%3Apr+label%3Arelease%2Fv1.8.1+is%3Aclosed
>
> > >
>
> > > Please download the source files and review this release candidate:
>
> > > - Download the source package, verify shasum and asc
>
> > > - Follow the README.md to build and run the Pulsar Node.js client.
>
> > >
>
> > > The release candidate package has been published to the npm
>
> > > registry:https://www.npmjs.com/package/pulsar-client/v/1.8.1-rc.1
>
> > > You can install it by `npm i pulsar-client@1.8.1-rc.1
>
> > > --pulsar_binary_host_mirror=
>
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>
> 
>
> > > 
>
> > > and verify the package.
>
> > >
>
> > > The vote will be open for at least 72 hours. It is adopted by majority
>
> > > approval, with at least 3 PMC affirmative votes.
>
> > >
>
> > > Source files:
>
> > >
>
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.8.1-rc.1/
>
> > >
>
> > > Pulsar's KEYS file containing PGP keys we use to sign the
>
> > > release:https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
> > >
>
> > > SHA-512 checksum:
>
> > >
>
> > >
>
>
> ed89b4ad467d3cb75ed37096b35d91b872cd93d36cd953512fc7edcb75dbac5162592f6f51b5ab08f26b3dca1c57a3d3fe7a5e4f109551c66943a5b09392d51a
>
> > >