Hi Dave, > This would be a commitment by the Pulsar PMC to support third party code.
Well. I think you mean the third party code from StreamNative is not as reliable as etcd or RocksDB. I agree. The current problem is that the MetadataStore is not pluggable. What if a personal contributor contributes a metadata store to an external metadata store system? What commitment should be made? Or do you intend to reject the proposal because it's not reliable because he might not maintain the external metadata store anymore? Thanks, Yunze On Sun, Feb 4, 2024 at 3:23 AM Dave Fisher <w...@apache.org> wrote: > > Hi - > > What? A cascade of agreement without questions? Seems like this has been > discussed elsewhere? > > I have concerns since with this PIP we are making changes to support a third > party library. > > 1. Have people read the PIP text? I see some typos which could be fixed. > Simple things nothing objectionable. > > 2. This would be a commitment by the Pulsar PMC to support third party code. > What commitment is there by StreamNative to keep the two Oxia repositories > open source? Relevant because I’ve heard that some StreamNative protocol > handlers have been archived. I know it’s not completely analogous, but if we > are making changes to Pulsar then an explicit commitment is best. > > Best, > Dave > > > On Jan 31, 2024, at 3:57 PM, Matteo Merli <matteo.me...@gmail.com> wrote: > > > > https://github.com/apache/pulsar/pull/22009 > > > > =========== > > > > # PIP-335: Supporty Oxia metadata store plugin > > > > # Motivation > > > > Oxia is a scalable metadata store and coordination system that can be used > > as the core infrastructure > > to build large scale distributed systems. > > > > Oxia was created with the primary goal of providing an alternative Pulsar > > to replace ZooKeeper as the > > long term preferred metadata store, overcoming all the current limitations > > in terms of metadata > > access throughput and data set size. > > > > # Goals > > > > Add a Pulsar MetadataStore plugin that uses Oxia client SDK. > > > > Users will be able to start a Pulsar cluster using just Oxia, without any > > ZooKeeper involved. > > > > ## Not in Scope > > > > It's not in the scope of this proposal to change any default behavior or > > configuration of Pulsar. > > > > # Detailed Design > > > > ## Design & Implementation Details > > > > Oxia semantics and client SDK were already designed with Pulsar and > > MetadataStore plugin API in mind, so > > there is not much integration work that needs to be done here. > > > > Just a few notes: > > 1. Oxia client already provides support for transparent batching of read > > and write operations, > > so there will be no use of the batching logic in > > `AbstractBatchedMetadataStore` > > 2. Oxia does not treat keys as a walkable file-system like interface, with > > directories and files. Instead > > all the keys are independent. Though Oxia sorting of keys is aware of > > '/' and provides efficient key > > range scanning operations to identify the first level children of a > > given key > > 3. Oxia, unlike ZooKeeper, doesn't require the parent path of a key to > > exist. eg: we can create `/a/b/c` key > > without `/a/b` and `/a` existing. > > In the Pulsar integration for Oxia we're forcing to create all parent > > keys when they are not there. This > > is due to several places in BookKeeper access where it does not create > > the parent keys, though it will > > later make `getChildren()` operations on the parents. > > > > ## Other notes > > > > Unlike in the ZooKeeper implementation, the notification of events is > > guaranteed in Oxia, because the Oxia > > client SDK will use the transaction offset after server reconnections and > > session restarted events. This > > will ensure that brokers cache will always be properly invalidated. We will > > then be able to remove the > > current 5minutes automatic cache refresh which is in place to prevent the > > ZooKeeper missed watch issue. > > > > # Links > > > > Oxia: https://github.com/streamnative/oxia > > Oxia Java Client SDK: https://github.com/streamnative/oxia-java > > > > > > -- > > Matteo Merli > > <matteo.me...@gmail.com> >
