315157973 opened a new pull request #10729: URL: https://github.com/apache/pulsar/pull/10729
### Motivation Pulsar persists the individual acknowledgments into Ledger for each subscription. `acknowledgment hole` uses guava range to record. Now a single entry is used to record these holes, but a single entry can not stores more than 5MB data. Therefore, this PIP stores these holes in different entries separately. And use LRU to ensure low memory usage. In this way, we can support almost unlimited `acknowledgment hole`. Doc: https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit ### Modifications recover: 1. Added API `recoverFromMultiEntry` 2. The old logic is abstract as `recoverFromSingleEntry` persist position to ledger： 1. Add API `persistPositionToMultiEntry` and `saveMarker` 2. The old logic is abstract as `persistPositionToSingleEntry` when creating new ledger: 1. Add API `copyLruEntriesToNewLedger` 2. The old logic remains the same LRU: 1. Add class RangeSetWrapper. Responsible for loading and eliminating of LRU data . 2. The loading of LRU data is synchronous, and elimination of LRU data is asynchronous, with `RangeSetWrapper#LruTask` ### Verifying this change persist position to ledger has 3 chain: ``` internalAsyncMarkDelete + NoLedger -> startCreatingNewMetadataLedger -> createNewMetadataLedgerAndSwitch -> doCreateNewMetadataLedger -> persistPositionToLedger -> switchToNewLedger -> flushPendingMarkDeletes -> internalMarkDelete -> persistPositionToLedger -> remove individualDeletedMessages ``` ``` internalAsyncMarkDelete + Ledger is Open -> internalMarkDelete -> persistPositionToLedger -> remove individualDeletedMessages ``` ``` persistPositionWhenClosing -> persistPositionToLedger -> close ledger ``` create new ledger has 1 chain: ``` initialize -> createNewMetadataLedgerAndSwitch -> doCreateNewMetadataLedger -> switchToNewLedger -> flushPendingMarkDeletes -> internalMarkDelete -> persistPositionToLedger -> remove individualDeletedMessages ``` recover has 1 chain: ``` initializeCursors -> recover -> recoverFromLedger -> recoverFromMultiEntry ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
