Re: [Discuss] Release Pulsar C++ Client 3.5.0

[Zike Yang]

+1

Thanks for driving the release.

BR,
Zike Yang

On Wed, Feb 7, 2024 at 2:20 PM Yunze Xu  wrote:
>
> I would like to propose releasing the Pulsar C++ Client 3.5.0.
>
> It has been about 2 months since the last release (3.4.2). There have
> been many new features and bug fixes since then. It's time to release
> a new version. I will start the release once
> https://github.com/apache/pulsar-client-cpp/pull/401 is ready.
>
> Please let me know if you have any PRs that need to be included in 3.5.0.
>
> Thanks,
> Yunze

[Discuss] Release Pulsar C++ Client 3.5.0

[Yunze Xu]

I would like to propose releasing the Pulsar C++ Client 3.5.0.

It has been about 2 months since the last release (3.4.2). There have
been many new features and bug fixes since then. It's time to release
a new version. I will start the release once
https://github.com/apache/pulsar-client-cpp/pull/401 is ready.

Please let me know if you have any PRs that need to be included in 3.5.0.

Thanks,
Yunze

Re: [DISCUSS] Deletion of Current Ledger upon Rollover

[Xiangying Meng]

>The problem is that if the ledger is deleted, the next time a client
produces a message, a new ledger needs to be opened. This is an operation
that may take some time, disrupting latency.

Hi Enrico, I think there may be some misunderstandings. If the state of the
managerLedger is `ClosedLedger`, the current ledger is closed and cannot be
written to. So when a client sends new messages, it must open a new ledger.
This will cause latency regardless of whether the current ledger is
deleted, since the current ledger is already closed.

In the discussion of the PR[1], we found that there are different behaviors
in Pulsar after ledger rollover.
1. If the rollover is triggered in the process of adding an entry, a new
ledger will be created depending on whether there are pending write
operations.
2. In other cases, a new ledger will be created directly regardless of
whether there are pending write operations.

In the latest implementation of the PR, for the first situation, a new
ledger will be created directly even if there are no pending write
operations. Of course, this will increase the overhead of the bookkeeper.
But for normal production and consumption scenarios, there should be no
situation where the new ledger is not written to for a long time. Even for
inactive topics, users can handle this situation according to the
inactive_topic_policies.

[1] - https://github.com/apache/pulsar/pull/22034

On Tue, Feb 6, 2024 at 9:55 PM Enrico Olivelli  wrote:

> Xiangying
>
> Il giorno mar 6 feb 2024 alle ore 13:01 Xiangying Meng
>  ha scritto:
> >
> > Dear Community,
> >
> > I hope this message finds you well. I am writing to discuss a
> modification
> > to the behavior of deleting the current ledger. As you may know, in
> Pulsar,
> > the current ledger cannot be deleted because it may still be written to.
> > However, there is an exception. When the current ledger is rolled over,
> but
> > no new messages are written, the current ledger does not change. In this
> > case, the current ledger will not be written to, but it is also not
> deleted.
>
> I understand the problem and I was surprised about it the first time I saw
> it.
> The problem is that if you delete the ledger the next time a client
> produces a message the broken
> must open a new ledger and this is an operation that may take some
> time, disrupting latency.
>
> It is a trade-off, I know, in production you usually don't need to
> release the disk space.
> Maybe you have a use case in which you write a lot to a topic, then
> you stop writing ?
>
> Maybe you could "unload" the topic, and that will force a ledger
> rollover (with an impact on latency)
>
> Enrico
>
> >
> > This can be confusing for users, especially when they configure
> > `managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
> > They expect the current ledger to roll over and then be deleted after
> > `managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
> > However, in reality, while the current ledger does rollover, it is not
> > deleted.
> >
> > The purpose of this discussion is to consider deleting the current ledger
> > when it is rolled over. The specific implementation can be found at
> > https://github.com/apache/pulsar/pull/22034.
> >
> > Looking forward to a productive discussion.
> >
> > Best Regards,
> >
> > xiangying
>

[ANNOUNCE] Apache Pulsar Node.js client 1.10.0 released

[Baodi Shi]

The Apache Pulsar team is proud to announce Apache Pulsar Node.js
client version 1.10.0.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management for
subscribers, and cross-datacenter replication.

For Pulsar Node.js client release details and downloads, visit:
https://www.npmjs.com/package/pulsar-client

Release Notes are at:
https://github.com/apache/pulsar-client-node/releases

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Re: [VOTE] PIP-335: Oxia metadata support

[Dezhi Liu]

+1 (non-binding)

Thanks,
Dezhi

On 2024/02/05 20:40:02 Matteo Merli wrote:
> https://github.com/apache/pulsar/pull/22009
> 
> -
> 
> # PIP-335: Support Oxia metadata store plugin
> 
> # Motivation
> 
> Oxia is a scalable metadata store and coordination system that can be used
> as the core infrastructure
> to build large scale distributed systems.
> 
> Oxia was created with the primary goal of providing an alternative Pulsar
> to replace ZooKeeper as the
> long term preferred metadata store, overcoming all the current limitations
> in terms of metadata
> access throughput and data set size.
> 
> # Goals
> 
> Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> 
> Users will be able to start a Pulsar cluster using just Oxia, without any
> ZooKeeper involved.
> 
> ## Not in Scope
> 
> It's not in the scope of this proposal to change any default behavior or
> configuration of Pulsar.
> 
> # Detailed Design
> 
> ## Design & Implementation Details
> 
> Oxia semantics and client SDK were already designed with Pulsar and
> MetadataStore plugin API in mind, so
> there is not much integration work that needs to be done here.
> 
> Just few notes:
>  1. Oxia client already provides support for transparent batching of read
> and write operations,
> so there will be no use of the batching logic in
> `AbstractBatchedMetadataStore`
>  2. Oxia does not treat keys as a walkable file-system like interface, with
> directories and files. Instead
> all the keys are independent. Though Oxia sorting of keys is aware of
> '/' and provides efficient key
> range scanning operations to identify the first level children of a
> given key
>  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> exist. eg: we can create `/a/b/c` key
> without `/a/b` and `/a` existing.
> In the Pulsar integration for Oxia we're forcing to create all parent
> keys when they are not there. This
> is due to several places in BookKeeper access where it does not create
> the parent keys, though it will
> later make `getChildren()` operations on the parents.
> 
> ## Other notes
> 
> Unlike in the ZooKeeper implementation, the notification of events is
> guaranteed in Oxia, because the Oxia
> client SDK will use the transaction offset after server reconnections and
> session restarted events. This
> will ensure that brokers cache will always be properly invalidated. We will
> then be able to remove the
> current 5minutes automatic cache refresh which is in place to prevent the
> ZooKeeper missed watch issue.
> 
> # Links
> 
> Oxia: https://github.com/streamnative/oxia
> Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> 
> 
> --
> Matteo Merli
> 
> 

Re: [VOTE] PIP-335: Oxia metadata support

[Kai Wang]

+1 (non-binding)

Thanks,
Kai

HI - Introducing myself to the community, exploring/adopting Pulsar!

[Austin Bennett]

Hi Pulsar Devs,

This is my first message to this list, but I've been subscribed,
following/reading this list for a few years.  Haven't seen many of these
types of messages, but imagine it is not unwelcomed [ often good for
developers to get a feel for potential colleagues and users ].

I am now looking to get started to really evaluate adopting Pulsar through
my role as Architect at ChartBoost , an adtech
company.  Most relevant, by way of introduction and competencies, I have
been working on Google Cloud Platform deeply for years, and am a committer
for Apache Beam [ mentioning since relevant to 'streaming' ;-p ] and am
involved in a number of other OSS projects in various capacities.

I expect that we'll rely on StreamNative to manage our Pulsar operations.
Though  utilizing and likely extending the integrations with core GCP
tooling will certainly be of interest to us.

Assuming eval goes as well, I'll be on the lookout to find ways to help as
sensible and useful, contributing likely by way of any lacking
documentation, sharing the good word of uses/successes [ writeups, talks,
etc ], and addressing lacking features when needed.

Cheers,
Austin

[DISCUSS] PIP-337: SSL Factory Plugin to customize SSL Context and SSL Engine generation

[Apurva Telang]

Hello All,

I have submitted a PIP to add a SSL Factory Plugin to customize the
generation of SSL Context and SSL Engine.

Motivation:
Apache Pulsar's TLS encryption configuration is not pluggable. It only
supports file-based certificates. This makes Pulsar difficult to adopt for
organizations that require loading TLS certificates by other mechanisms.

Goals:

   1. Provide a mechanism to plugin a custom SSL Factory that can generate
   SSL Context and SSL Engine.
   2. Simplify the Pulsar code base to universally use
   javax.net.ssl.SSLContext and reduce the amount of code required to build
   and configure the SSL context taking into consideration backwards
   compatibility.

PIP link: https://github.com/apache/pulsar/pull/22016

Please can the community review the PIP and provide your feedback.

--
Best regards,
Apurva Telang.

Re: [VOTE] PIP-335: Oxia metadata support

[Apurva Telang]

+1 (non-binding)

On Tue, Feb 6, 2024 at 7:29 AM Zixuan Liu  wrote:

> +1(non-binding)
>
> Thanks,
> Zixuan
>
> Zike Yang  于2024年2月6日周二 19:29写道：
>
> > +1
> >
> > Thanks,
> > Zike Yang
> >
> > On Tue, Feb 6, 2024 at 6:58 PM Enrico Olivelli 
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > Enrico
> > >
> > > Il giorno mar 6 feb 2024 alle ore 11:36 houxiaoyu
> > >  ha scritto:
> > > >
> > > > +1 nonbinding
> > > >
> > > > Thanks,
> > > > Xiaoyu Hou
> > > >
> > > > 太上玄元道君  于2024年2月6日周二 15:59写道：
> > > > >
> > > > > +1 nonbinding
> > > > >
> > > > > Matteo Merli 于2024年2月6日 周二04:40写道：
> > > > >
> > > > > > https://github.com/apache/pulsar/pull/22009
> > > > > >
> > > > > > -
> > > > > >
> > > > > > # PIP-335: Support Oxia metadata store plugin
> > > > > >
> > > > > > # Motivation
> > > > > >
> > > > > > Oxia is a scalable metadata store and coordination system that
> can
> > be used
> > > > > > as the core infrastructure
> > > > > > to build large scale distributed systems.
> > > > > >
> > > > > > Oxia was created with the primary goal of providing an
> alternative
> > Pulsar
> > > > > > to replace ZooKeeper as the
> > > > > > long term preferred metadata store, overcoming all the current
> > limitations
> > > > > > in terms of metadata
> > > > > > access throughput and data set size.
> > > > > >
> > > > > > # Goals
> > > > > >
> > > > > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> > > > > >
> > > > > > Users will be able to start a Pulsar cluster using just Oxia,
> > without any
> > > > > > ZooKeeper involved.
> > > > > >
> > > > > > ## Not in Scope
> > > > > >
> > > > > > It's not in the scope of this proposal to change any default
> > behavior or
> > > > > > configuration of Pulsar.
> > > > > >
> > > > > > # Detailed Design
> > > > > >
> > > > > > ## Design & Implementation Details
> > > > > >
> > > > > > Oxia semantics and client SDK were already designed with Pulsar
> and
> > > > > > MetadataStore plugin API in mind, so
> > > > > > there is not much integration work that needs to be done here.
> > > > > >
> > > > > > Just few notes:
> > > > > >  1. Oxia client already provides support for transparent batching
> > of read
> > > > > > and write operations,
> > > > > > so there will be no use of the batching logic in
> > > > > > `AbstractBatchedMetadataStore`
> > > > > >  2. Oxia does not treat keys as a walkable file-system like
> > interface, with
> > > > > > directories and files. Instead
> > > > > > all the keys are independent. Though Oxia sorting of keys is
> > aware of
> > > > > > '/' and provides efficient key
> > > > > > range scanning operations to identify the first level
> children
> > of a
> > > > > > given key
> > > > > >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a
> > key to
> > > > > > exist. eg: we can create `/a/b/c` key
> > > > > > without `/a/b` and `/a` existing.
> > > > > > In the Pulsar integration for Oxia we're forcing to create
> all
> > parent
> > > > > > keys when they are not there. This
> > > > > > is due to several places in BookKeeper access where it does
> > not create
> > > > > > the parent keys, though it will
> > > > > > later make `getChildren()` operations on the parents.
> > > > > >
> > > > > > ## Other notes
> > > > > >
> > > > > > Unlike in the ZooKeeper implementation, the notification of
> events
> > is
> > > > > > guaranteed in Oxia, because the Oxia
> > > > > > client SDK will use the transaction offset after server
> > reconnections and
> > > > > > session restarted events. This
> > > > > > will ensure that brokers cache will always be properly
> > invalidated. We will
> > > > > > then be able to remove the
> > > > > > current 5minutes automatic cache refresh which is in place to
> > prevent the
> > > > > > ZooKeeper missed watch issue.
> > > > > >
> > > > > > # Links
> > > > > >
> > > > > > Oxia: https://github.com/streamnative/oxia
> > > > > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Matteo Merli
> > > > > > 
> > > > > >
> >
>


-- 
Best regards,
Apurva Telang.

Re: [VOTE] PIP-335: Oxia metadata support

[Zixuan Liu]

+1(non-binding)

Thanks,
Zixuan

Zike Yang  于2024年2月6日周二 19:29写道：

> +1
>
> Thanks,
> Zike Yang
>
> On Tue, Feb 6, 2024 at 6:58 PM Enrico Olivelli 
> wrote:
> >
> > +1 (binding)
> >
> > Enrico
> >
> > Il giorno mar 6 feb 2024 alle ore 11:36 houxiaoyu
> >  ha scritto:
> > >
> > > +1 nonbinding
> > >
> > > Thanks,
> > > Xiaoyu Hou
> > >
> > > 太上玄元道君  于2024年2月6日周二 15:59写道：
> > > >
> > > > +1 nonbinding
> > > >
> > > > Matteo Merli 于2024年2月6日 周二04:40写道：
> > > >
> > > > > https://github.com/apache/pulsar/pull/22009
> > > > >
> > > > > -
> > > > >
> > > > > # PIP-335: Support Oxia metadata store plugin
> > > > >
> > > > > # Motivation
> > > > >
> > > > > Oxia is a scalable metadata store and coordination system that can
> be used
> > > > > as the core infrastructure
> > > > > to build large scale distributed systems.
> > > > >
> > > > > Oxia was created with the primary goal of providing an alternative
> Pulsar
> > > > > to replace ZooKeeper as the
> > > > > long term preferred metadata store, overcoming all the current
> limitations
> > > > > in terms of metadata
> > > > > access throughput and data set size.
> > > > >
> > > > > # Goals
> > > > >
> > > > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> > > > >
> > > > > Users will be able to start a Pulsar cluster using just Oxia,
> without any
> > > > > ZooKeeper involved.
> > > > >
> > > > > ## Not in Scope
> > > > >
> > > > > It's not in the scope of this proposal to change any default
> behavior or
> > > > > configuration of Pulsar.
> > > > >
> > > > > # Detailed Design
> > > > >
> > > > > ## Design & Implementation Details
> > > > >
> > > > > Oxia semantics and client SDK were already designed with Pulsar and
> > > > > MetadataStore plugin API in mind, so
> > > > > there is not much integration work that needs to be done here.
> > > > >
> > > > > Just few notes:
> > > > >  1. Oxia client already provides support for transparent batching
> of read
> > > > > and write operations,
> > > > > so there will be no use of the batching logic in
> > > > > `AbstractBatchedMetadataStore`
> > > > >  2. Oxia does not treat keys as a walkable file-system like
> interface, with
> > > > > directories and files. Instead
> > > > > all the keys are independent. Though Oxia sorting of keys is
> aware of
> > > > > '/' and provides efficient key
> > > > > range scanning operations to identify the first level children
> of a
> > > > > given key
> > > > >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a
> key to
> > > > > exist. eg: we can create `/a/b/c` key
> > > > > without `/a/b` and `/a` existing.
> > > > > In the Pulsar integration for Oxia we're forcing to create all
> parent
> > > > > keys when they are not there. This
> > > > > is due to several places in BookKeeper access where it does
> not create
> > > > > the parent keys, though it will
> > > > > later make `getChildren()` operations on the parents.
> > > > >
> > > > > ## Other notes
> > > > >
> > > > > Unlike in the ZooKeeper implementation, the notification of events
> is
> > > > > guaranteed in Oxia, because the Oxia
> > > > > client SDK will use the transaction offset after server
> reconnections and
> > > > > session restarted events. This
> > > > > will ensure that brokers cache will always be properly
> invalidated. We will
> > > > > then be able to remove the
> > > > > current 5minutes automatic cache refresh which is in place to
> prevent the
> > > > > ZooKeeper missed watch issue.
> > > > >
> > > > > # Links
> > > > >
> > > > > Oxia: https://github.com/streamnative/oxia
> > > > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> > > > >
> > > > >
> > > > > --
> > > > > Matteo Merli
> > > > > 
> > > > >
>

Re: [DISCUSS] Deletion of Current Ledger upon Rollover

[Enrico Olivelli]

Xiangying

Il giorno mar 6 feb 2024 alle ore 13:01 Xiangying Meng
 ha scritto:
>
> Dear Community,
>
> I hope this message finds you well. I am writing to discuss a modification
> to the behavior of deleting the current ledger. As you may know, in Pulsar,
> the current ledger cannot be deleted because it may still be written to.
> However, there is an exception. When the current ledger is rolled over, but
> no new messages are written, the current ledger does not change. In this
> case, the current ledger will not be written to, but it is also not deleted.

I understand the problem and I was surprised about it the first time I saw it.
The problem is that if you delete the ledger the next time a client
produces a message the broken
must open a new ledger and this is an operation that may take some
time, disrupting latency.

It is a trade-off, I know, in production you usually don't need to
release the disk space.
Maybe you have a use case in which you write a lot to a topic, then
you stop writing ?

Maybe you could "unload" the topic, and that will force a ledger
rollover (with an impact on latency)

Enrico

>
> This can be confusing for users, especially when they configure
> `managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
> They expect the current ledger to roll over and then be deleted after
> `managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
> However, in reality, while the current ledger does rollover, it is not
> deleted.
>
> The purpose of this discussion is to consider deleting the current ledger
> when it is rolled over. The specific implementation can be found at
> https://github.com/apache/pulsar/pull/22034.
>
> Looking forward to a productive discussion.
>
> Best Regards,
>
> xiangying

[DISCUSS] Deletion of Current Ledger upon Rollover

[Xiangying Meng]

Dear Community,

I hope this message finds you well. I am writing to discuss a modification
to the behavior of deleting the current ledger. As you may know, in Pulsar,
the current ledger cannot be deleted because it may still be written to.
However, there is an exception. When the current ledger is rolled over, but
no new messages are written, the current ledger does not change. In this
case, the current ledger will not be written to, but it is also not deleted.

This can be confusing for users, especially when they configure
`managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
They expect the current ledger to roll over and then be deleted after
`managedLedgerMaxLedgerRolloverTimeMinutes` and `retentionTimeInMinutes`.
However, in reality, while the current ledger does rollover, it is not
deleted.

The purpose of this discussion is to consider deleting the current ledger
when it is rolled over. The specific implementation can be found at
https://github.com/apache/pulsar/pull/22034.

Looking forward to a productive discussion.

Best Regards,

xiangying

Re: [VOTE] Pulsar Node.js Client Release 1.10.0 Candidate 1

[Baodi Shi]

Closing this vote by 3 binding +1s
- Yunze
- Yubiao
- Massahiro


Thanks
Baodi Shi


On Feb 6, 2024 at 10:13:41, Yubiao Feng 
wrote:

> +1 (binding)
>
> - verified checksums and signatures
> - confirmed that the build was successful
> - ran producer/consumer
>
> Thanks
> Yubaio Feng
>
> On Mon, Jan 29, 2024 at 6:11 PM Baodi Shi  wrote:
>
> Hi everyone,
>
>
> This is the first release candidate for Apache Pulsar Node.js client,
>
> version 1.10.0.
>
>
> It fixes the following issues:
>
> https://github.com/apache/pulsar-client-node/milestone/15?closed=1
>
>
> Please download the source files and review this release candidate:
>
> - Download the source package, verify shasum and asc
>
> - Follow the README.md to build and run the Pulsar Node.js client.
>
>
> The release candidate package has been published to the npm registry:
>
> https://www.npmjs.com/package/pulsar-client/v/1.10.0-rc.1
>
> You can install it by `npm i pulsar-client@1.10.0-rc.1
>
> --pulsar_binary_host_mirror=
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/`
>
>  and
>
> verify the package.
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
>
> approval, with at least 3 PMC affirmative votes.
>
>
> Source files:
>
>
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-node/pulsar-client-node-1.10.0-rc.1/
>
>
> Pulsar's KEYS file containing PGP keys we use to sign the release:
>
> https://downloads.apache.org/pulsar/KEYS
>
>
> SHA-512 checksum:
>
>
>
> 3e2c76ff8340597129eec1ab2a9cdbdd48fb162257426c474bd637bb02a45b49011e6b31e08679022f18d700e73f49cbb8adc59f75e345db9220a2b755eccac4
>
> ./apache-pulsar-client-node-1.10.0.tar.gz
>
>
> The tag to be voted upon:
>
> v1.10.0-rc.1
>
> https://github.com/apache/pulsar-client-node/releases/tag/v1.10.0-rc.1
>
>
> Please review and vote on the release candidate #1 for the version 1.10.0,
>
> as follows:
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
>

Re: [VOTE] PIP-335: Oxia metadata support

[Zike Yang]

+1

Thanks,
Zike Yang

On Tue, Feb 6, 2024 at 6:58 PM Enrico Olivelli  wrote:
>
> +1 (binding)
>
> Enrico
>
> Il giorno mar 6 feb 2024 alle ore 11:36 houxiaoyu
>  ha scritto:
> >
> > +1 nonbinding
> >
> > Thanks,
> > Xiaoyu Hou
> >
> > 太上玄元道君  于2024年2月6日周二 15:59写道：
> > >
> > > +1 nonbinding
> > >
> > > Matteo Merli 于2024年2月6日 周二04:40写道：
> > >
> > > > https://github.com/apache/pulsar/pull/22009
> > > >
> > > > -
> > > >
> > > > # PIP-335: Support Oxia metadata store plugin
> > > >
> > > > # Motivation
> > > >
> > > > Oxia is a scalable metadata store and coordination system that can be 
> > > > used
> > > > as the core infrastructure
> > > > to build large scale distributed systems.
> > > >
> > > > Oxia was created with the primary goal of providing an alternative 
> > > > Pulsar
> > > > to replace ZooKeeper as the
> > > > long term preferred metadata store, overcoming all the current 
> > > > limitations
> > > > in terms of metadata
> > > > access throughput and data set size.
> > > >
> > > > # Goals
> > > >
> > > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> > > >
> > > > Users will be able to start a Pulsar cluster using just Oxia, without 
> > > > any
> > > > ZooKeeper involved.
> > > >
> > > > ## Not in Scope
> > > >
> > > > It's not in the scope of this proposal to change any default behavior or
> > > > configuration of Pulsar.
> > > >
> > > > # Detailed Design
> > > >
> > > > ## Design & Implementation Details
> > > >
> > > > Oxia semantics and client SDK were already designed with Pulsar and
> > > > MetadataStore plugin API in mind, so
> > > > there is not much integration work that needs to be done here.
> > > >
> > > > Just few notes:
> > > >  1. Oxia client already provides support for transparent batching of 
> > > > read
> > > > and write operations,
> > > > so there will be no use of the batching logic in
> > > > `AbstractBatchedMetadataStore`
> > > >  2. Oxia does not treat keys as a walkable file-system like interface, 
> > > > with
> > > > directories and files. Instead
> > > > all the keys are independent. Though Oxia sorting of keys is aware 
> > > > of
> > > > '/' and provides efficient key
> > > > range scanning operations to identify the first level children of a
> > > > given key
> > > >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> > > > exist. eg: we can create `/a/b/c` key
> > > > without `/a/b` and `/a` existing.
> > > > In the Pulsar integration for Oxia we're forcing to create all 
> > > > parent
> > > > keys when they are not there. This
> > > > is due to several places in BookKeeper access where it does not 
> > > > create
> > > > the parent keys, though it will
> > > > later make `getChildren()` operations on the parents.
> > > >
> > > > ## Other notes
> > > >
> > > > Unlike in the ZooKeeper implementation, the notification of events is
> > > > guaranteed in Oxia, because the Oxia
> > > > client SDK will use the transaction offset after server reconnections 
> > > > and
> > > > session restarted events. This
> > > > will ensure that brokers cache will always be properly invalidated. We 
> > > > will
> > > > then be able to remove the
> > > > current 5minutes automatic cache refresh which is in place to prevent 
> > > > the
> > > > ZooKeeper missed watch issue.
> > > >
> > > > # Links
> > > >
> > > > Oxia: https://github.com/streamnative/oxia
> > > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > 
> > > >

Re: [VOTE] PIP-335: Oxia metadata support

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno mar 6 feb 2024 alle ore 11:36 houxiaoyu
 ha scritto:
>
> +1 nonbinding
>
> Thanks,
> Xiaoyu Hou
>
> 太上玄元道君  于2024年2月6日周二 15:59写道：
> >
> > +1 nonbinding
> >
> > Matteo Merli 于2024年2月6日 周二04:40写道：
> >
> > > https://github.com/apache/pulsar/pull/22009
> > >
> > > -
> > >
> > > # PIP-335: Support Oxia metadata store plugin
> > >
> > > # Motivation
> > >
> > > Oxia is a scalable metadata store and coordination system that can be used
> > > as the core infrastructure
> > > to build large scale distributed systems.
> > >
> > > Oxia was created with the primary goal of providing an alternative Pulsar
> > > to replace ZooKeeper as the
> > > long term preferred metadata store, overcoming all the current limitations
> > > in terms of metadata
> > > access throughput and data set size.
> > >
> > > # Goals
> > >
> > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> > >
> > > Users will be able to start a Pulsar cluster using just Oxia, without any
> > > ZooKeeper involved.
> > >
> > > ## Not in Scope
> > >
> > > It's not in the scope of this proposal to change any default behavior or
> > > configuration of Pulsar.
> > >
> > > # Detailed Design
> > >
> > > ## Design & Implementation Details
> > >
> > > Oxia semantics and client SDK were already designed with Pulsar and
> > > MetadataStore plugin API in mind, so
> > > there is not much integration work that needs to be done here.
> > >
> > > Just few notes:
> > >  1. Oxia client already provides support for transparent batching of read
> > > and write operations,
> > > so there will be no use of the batching logic in
> > > `AbstractBatchedMetadataStore`
> > >  2. Oxia does not treat keys as a walkable file-system like interface, 
> > > with
> > > directories and files. Instead
> > > all the keys are independent. Though Oxia sorting of keys is aware of
> > > '/' and provides efficient key
> > > range scanning operations to identify the first level children of a
> > > given key
> > >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> > > exist. eg: we can create `/a/b/c` key
> > > without `/a/b` and `/a` existing.
> > > In the Pulsar integration for Oxia we're forcing to create all parent
> > > keys when they are not there. This
> > > is due to several places in BookKeeper access where it does not create
> > > the parent keys, though it will
> > > later make `getChildren()` operations on the parents.
> > >
> > > ## Other notes
> > >
> > > Unlike in the ZooKeeper implementation, the notification of events is
> > > guaranteed in Oxia, because the Oxia
> > > client SDK will use the transaction offset after server reconnections and
> > > session restarted events. This
> > > will ensure that brokers cache will always be properly invalidated. We 
> > > will
> > > then be able to remove the
> > > current 5minutes automatic cache refresh which is in place to prevent the
> > > ZooKeeper missed watch issue.
> > >
> > > # Links
> > >
> > > Oxia: https://github.com/streamnative/oxia
> > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >

Re: [DISCUSS] PIP-334:Create a new admin API: pck (Pulsar Consistency Checker)

[Jie crossover]

Sure, that's a good suggestion.
Maybe we can refer to the implementation of pulsar-perf.
-- 
Best Regards！
crossoverJie


Haiting Jiang  于2024年2月6日周二 11:38写道：

> The motivation LGTM.
> But I am not sure if it's possible to implement at client side,
> without adding a new admin API?
> My main concern is this method may run for quite a while. And it's not
> easy to see the progress if it's implemented on the server.
>
> Thanks,
> Haiting
>
> On Tue, Jan 30, 2024 at 6:50 PM Jie crossover 
> wrote:
> >
> > Hi dev,
> > I proposed a PIP: https://github.com/apache/pulsar/pull/21988
> >
> > Sometimes we need to check if there are any orphan ledgers in the
> > bookkeeper.
> > So I'm planning to create an admin API.
> >
> > Specific implementation logic is referred to in this link
> > <
> https://docs.streamnative.io/platform/pck-tutorial#find-orphan-ledgers-from-bookie
> >
> > .
> >
> > Please take a look and give your feedback.
> > --
> > Best Regards！
> > crossoverJie
>

Re: [VOTE] PIP-335: Oxia metadata support

[houxiaoyu]

+1 nonbinding

Thanks,
Xiaoyu Hou

太上玄元道君  于2024年2月6日周二 15:59写道：
>
> +1 nonbinding
>
> Matteo Merli 于2024年2月6日 周二04:40写道：
>
> > https://github.com/apache/pulsar/pull/22009
> >
> > -
> >
> > # PIP-335: Support Oxia metadata store plugin
> >
> > # Motivation
> >
> > Oxia is a scalable metadata store and coordination system that can be used
> > as the core infrastructure
> > to build large scale distributed systems.
> >
> > Oxia was created with the primary goal of providing an alternative Pulsar
> > to replace ZooKeeper as the
> > long term preferred metadata store, overcoming all the current limitations
> > in terms of metadata
> > access throughput and data set size.
> >
> > # Goals
> >
> > Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
> >
> > Users will be able to start a Pulsar cluster using just Oxia, without any
> > ZooKeeper involved.
> >
> > ## Not in Scope
> >
> > It's not in the scope of this proposal to change any default behavior or
> > configuration of Pulsar.
> >
> > # Detailed Design
> >
> > ## Design & Implementation Details
> >
> > Oxia semantics and client SDK were already designed with Pulsar and
> > MetadataStore plugin API in mind, so
> > there is not much integration work that needs to be done here.
> >
> > Just few notes:
> >  1. Oxia client already provides support for transparent batching of read
> > and write operations,
> > so there will be no use of the batching logic in
> > `AbstractBatchedMetadataStore`
> >  2. Oxia does not treat keys as a walkable file-system like interface, with
> > directories and files. Instead
> > all the keys are independent. Though Oxia sorting of keys is aware of
> > '/' and provides efficient key
> > range scanning operations to identify the first level children of a
> > given key
> >  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> > exist. eg: we can create `/a/b/c` key
> > without `/a/b` and `/a` existing.
> > In the Pulsar integration for Oxia we're forcing to create all parent
> > keys when they are not there. This
> > is due to several places in BookKeeper access where it does not create
> > the parent keys, though it will
> > later make `getChildren()` operations on the parents.
> >
> > ## Other notes
> >
> > Unlike in the ZooKeeper implementation, the notification of events is
> > guaranteed in Oxia, because the Oxia
> > client SDK will use the transaction offset after server reconnections and
> > session restarted events. This
> > will ensure that brokers cache will always be properly invalidated. We will
> > then be able to remove the
> > current 5minutes automatic cache refresh which is in place to prevent the
> > ZooKeeper missed watch issue.
> >
> > # Links
> >
> > Oxia: https://github.com/streamnative/oxia
> > Oxia Java Client SDK: https://github.com/streamnative/oxia-java
> >
> >
> > --
> > Matteo Merli
> > 
> >

Re: [VOTE] PIP-335: Oxia metadata support

[太上玄元道君]

+1 nonbinding

Matteo Merli 于2024年2月6日 周二04:40写道：

> https://github.com/apache/pulsar/pull/22009
>
> -
>
> # PIP-335: Support Oxia metadata store plugin
>
> # Motivation
>
> Oxia is a scalable metadata store and coordination system that can be used
> as the core infrastructure
> to build large scale distributed systems.
>
> Oxia was created with the primary goal of providing an alternative Pulsar
> to replace ZooKeeper as the
> long term preferred metadata store, overcoming all the current limitations
> in terms of metadata
> access throughput and data set size.
>
> # Goals
>
> Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
>
> Users will be able to start a Pulsar cluster using just Oxia, without any
> ZooKeeper involved.
>
> ## Not in Scope
>
> It's not in the scope of this proposal to change any default behavior or
> configuration of Pulsar.
>
> # Detailed Design
>
> ## Design & Implementation Details
>
> Oxia semantics and client SDK were already designed with Pulsar and
> MetadataStore plugin API in mind, so
> there is not much integration work that needs to be done here.
>
> Just few notes:
>  1. Oxia client already provides support for transparent batching of read
> and write operations,
> so there will be no use of the batching logic in
> `AbstractBatchedMetadataStore`
>  2. Oxia does not treat keys as a walkable file-system like interface, with
> directories and files. Instead
> all the keys are independent. Though Oxia sorting of keys is aware of
> '/' and provides efficient key
> range scanning operations to identify the first level children of a
> given key
>  3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to
> exist. eg: we can create `/a/b/c` key
> without `/a/b` and `/a` existing.
> In the Pulsar integration for Oxia we're forcing to create all parent
> keys when they are not there. This
> is due to several places in BookKeeper access where it does not create
> the parent keys, though it will
> later make `getChildren()` operations on the parents.
>
> ## Other notes
>
> Unlike in the ZooKeeper implementation, the notification of events is
> guaranteed in Oxia, because the Oxia
> client SDK will use the transaction offset after server reconnections and
> session restarted events. This
> will ensure that brokers cache will always be properly invalidated. We will
> then be able to remove the
> current 5minutes automatic cache refresh which is in place to prevent the
> ZooKeeper missed watch issue.
>
> # Links
>
> Oxia: https://github.com/streamnative/oxia
> Oxia Java Client SDK: https://github.com/streamnative/oxia-java
>
>
> --
> Matteo Merli
> 
>