merlimat opened a new pull request, #25693: URL: https://github.com/apache/pulsar/pull/25693
## Summary Sub-PIP of [PIP-460: Scalable Topics](https://github.com/apache/pulsar/blob/master/pip/pip-460.md). Proposes a transaction model for scalable topics that replaces the in-stream commit/abort marker mechanism with metadata-store-backed state, so transactions compose correctly with the segment lifecycle (splits, merges, sealed segments). The shape of the change: - **`MetadataTransactionBuffer`** — new `TransactionBuffer` implementation for `segment://` topics. Writes nothing to the topic's data stream; transactional state lives as records and secondary indexes in the metadata store. - **`MetadataPendingAckStore`** — new `PendingAckStore` implementation for `segment://` subscriptions. No sibling pending-ack topic. - **Transaction Coordinator V5** — parallel coordinator selected by v5 clients. Single CAS per end-txn; no system-topic log, no log compaction, no cold-start replay. Participants observe state transitions via metadata-store header watches — no broker-to-broker RPCs. - **`MetadataStore` extensions** — surface partition-key co-location, sequential keys, and secondary indexes with range-watch. Default backend (Oxia) maps to native primitives; ZooKeeper and others can implement them sub-optimally without affecting correctness. Existing `persistent://` topic behavior is unchanged. v4 transactions keep their current implementation byte-for-byte. The PIP is self-contained — full data model, end-to-end flows (publish, end-txn, dispatch, recovery, GC), backward-compat, and rejected alternatives are in the document. ## Test plan - [ ] Mailing list discussion - [ ] Mailing list vote -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
