merlimat opened a new pull request, #25724: URL: https://github.com/apache/pulsar/pull/25724
## Motivation Follow-up to #25723. Add atomic, multi-dimensional sequence keys to the `MetadataStore` API — same model as Oxia's native [sequence-keys](https://oxia-db.github.io/docs/features/sequence-keys) — and a companion subscription channel for change notifications. PIP-471 (transaction state via metadata) needs both: per-segment append-only logs (`/txn-op/<txnId>-<seq>`) where the server picks the suffix atomically, and a notification stream so each TB can apply only its own segment's events. The existing `Option.Sequential` (single-dim, +1 only, two-write fallback on Oxia) is preserved; this PR adds a richer primitive alongside it. ## Modifications **API**: - New `Option.SequenceKeysDeltas(List<Long> deltas)` — when present on `put(prefix, ...)`, the actual stored key becomes `<prefix>-<seq0:%020d>-<seq1:%020d>-...`. Each dimension increments atomically by its delta (first must be > 0, rest >= 0). The returned `Stat` carries the actual generated path. - New `MetadataStore.subscribeSequence(prefix, listener, opts) → AutoCloseable` — listener receives the latest assigned sequence key under `prefix` as new records appear. Multiple updates may collapse into a single emission with the highest sequence. Closing the handle unsubscribes. - `OptionsHelper.sequenceKeysDeltas(Set<Option>)` accessor. **Oxia (native)**: `OxiaMetadataStore` translates `Option.SequenceKeysDeltas` to `PutOption.SequenceKeysDeltas`; `subscribeSequence` delegates to `client.getSequenceUpdates`. Marks `supportsNativeSequenceKeys() == true` so the compat layer is bypassed. **Compatibility layer in `AbstractMetadataStore`** (used by LocalMemory, RocksDB, ZooKeeper, MockZooKeeper): - `put` intercepts `Option.SequenceKeysDeltas`, runs a CAS-retry loop on a sidecar counter document at `<prefix>__seq_counter__` (a fixed-width binary blob of `long`s), then recurses into `put` with the synthesized key. Synthesized keys match Oxia's byte format exactly. - `subscribeSequence` registers an internal `Notification` listener that filters `Created` events whose paths start with `<prefix>-`, tracks the highest path seen via CAS, and delivers monotonically. Closing the handle removes the listener. **Wrappers**: `DualMetadataStore` and `FaultInjectionMetadataStore` forward `subscribeSequence` to the wrapped store. ## Verifying this change Two new test classes: - `OxiaSequenceKeysTest` — 3-shard Oxia container, exercises the native path: single/multi-dim sequence puts, subscribe collapses to highest emission. - `SequenceKeysTest` — runs on every `BaseMetadataStoreTest` backend (ZK, Memory, RocksDB, Oxia, MockZooKeeper). Both compat and native paths must produce monotonically increasing keys with matching byte-format and deliver subscription updates. Local results: 15/15 cross-backend tests pass; full metadata test suite (`./gradlew :pulsar-metadata:test`) green. ## Does this pull request potentially affect one of the following parts: - [x] The public `MetadataStore` API — strictly additive: a new `Option` subtype and a new default method on `MetadataStore`. Existing callers and external implementations are unaffected. ## Matching PR in forked repository PR in forked repository: https://github.com/merlimat/pulsar/pull/new/mmerli/metadata-sequence-keys -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
