Ksnz commented on PR #24300: URL: https://github.com/apache/pulsar/pull/24300#issuecomment-3611783510
We could introduce an additional option similar to `replicatedSubscriptionsSnapshotFrequencyMillis`, but for message count – for example `replicatedSubscriptionsSnapshotFrequencyMessageGap` (the naming can be discussed later, along with the semantics of how the two properties interact), and send `ReplicatedSubscriptionsSnapshotRequest` not only based on time but also after a certain number of messages have been received. This way, we could achieve intervals between snapshots that are equal in terms of message count. However, I still don’t see how this would help when `markDelete` can be far beyond the horizon of the current cache. An additional option to solution would be to store the `ReplicatedSubscriptionSnapshotCache` somewhere in an **off-heap storage**, for example in **RocksDB**, evicting entries that are older than the `markDelete` horizon. This is the generic approach to keep both slow/delayed message consumers and fast backlog-tip consumers subscriptions aligned across clusters and to minimise inconsistency lag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
