This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push: new 1c5b4b8a0ca [feat] PIP-335: Oxia metadata plugin (#22009) 1c5b4b8a0ca is described below commit 1c5b4b8a0caf737c062d8e24879be3b592e0f5ed Author: Matteo Merli <mme...@apache.org> AuthorDate: Wed Feb 7 14:10:02 2024 -0800 [feat] PIP-335: Oxia metadata plugin (#22009) Co-authored-by: Zac Bentley <zbent...@users.noreply.github.com> --- pip/pip-335 Oxia metadata plugin.md | 51 +++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/pip/pip-335 Oxia metadata plugin.md b/pip/pip-335 Oxia metadata plugin.md new file mode 100644 index 00000000000..fdff8a85593 --- /dev/null +++ b/pip/pip-335 Oxia metadata plugin.md @@ -0,0 +1,51 @@ +# PIP-335: Support Oxia metadata store plugin + +# Motivation + +Oxia is a scalable metadata store and coordination system that can be used as the core infrastructure +to build large scale distributed systems. + +Oxia was created with the primary goal of providing an alternative Pulsar to replace ZooKeeper as the +long term preferred metadata store, overcoming all the current limitations in terms of metadata +access throughput and data set size. + +# Goals + +Add a Pulsar MetadataStore plugin that uses Oxia client SDK. + +Users will be able to start a Pulsar cluster using just Oxia, without any ZooKeeper involved. + +## Not in Scope + +It's not in the scope of this proposal to change any default behavior or configuration of Pulsar. + +# Detailed Design + +## Design & Implementation Details + +Oxia semantics and client SDK were already designed with Pulsar and MetadataStore plugin API in mind, so +there is not much integration work that needs to be done here. + +Just few notes: + 1. Oxia client already provides support for transparent batching of read and write operations, + so there will be no use of the batching logic in `AbstractBatchedMetadataStore` + 2. Oxia does not treat keys as a walkable file-system like interface, with directories and files. Instead + all the keys are independent. Though Oxia sorting of keys is aware of '/' and provides efficient key + range scanning operations to identify the first level children of a given key + 3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to exist. eg: we can create `/a/b/c` key + without `/a/b` and `/a` existing. + In the Pulsar integration for Oxia we're forcing to create all parent keys when they are not there. This + is due to several places in BookKeeper access where it does not create the parent keys, though it will + later make `getChildren()` operations on the parents. + +## Other notes + +Unlike in the ZooKeeper implementation, the notification of events is guaranteed in Oxia, because the Oxia +client SDK will use the transaction offset after server reconnections and session restarted events. This +will ensure that brokers cache will always be properly invalidated. We will then be able to remove the +current 5minutes automatic cache refresh which is in place to prevent the ZooKeeper missed watch issue. + +# Links + +Oxia: https://github.com/streamnative/oxia +Oxia Java Client SDK: https://github.com/streamnative/oxia-java