lhotari commented on PR #24300: URL: https://github.com/apache/pulsar/pull/24300#issuecomment-3611449379
> The main problem in my current project is that `markDelete` never advances on the delayed-message topic replica. In cluster A the `markDelete` cursor moves only rarely, at unpredictable moments, and over an unpredictable range. This causes the Cluster B replica backlog to grow. @Ksnz Yes, this seems like a problem case where the issue is caused by snapshot cache overflowing. > 2. **Equal distance intervals** (new strategy) — evict from the head. Requires extra code outside of `ReplicatedSubscriptionSnapshotCache,` because `Position` itself is not trusted source to calculate distance. There's existing org.apache.bookkeeper.mledger.ManagedLedger#getNumberOfEntries(com.google.common.collect.Range<org.apache.bookkeeper.mledger.Position>) available for the calculation. The reason why I think it's more useful than time based strategies is that the rate of consuming usually remains about the same while producing can have high bursts. > 3. **Increasing time intervals (from tail to head)** (new strategy) — evict from the middle, keep the head. > 4. **Increasing distance intervals (from tail to head)** (new strategy) — evict from the middle, keep the head. Again extra code. > 5. **Increasing time intervals (from head to tail)** (new strategy) — evict from the middle, keep the head. > 6. **Increasing distance intervals (from head to tail)** (new strategy) — evict from the middle, keep the head. Requires additional code . "evict from the middle" sounds great, but it's not practical due to the variable producing speed. It could work for topics where producing speed remains constant. > And an option to pick strategy per namespace. Ability to pick strategy on consumer configuration requires a cascade of code changes. Having too many options to pick from is not a great idea. It should be possible to solve this snapshot caching issue in a generic way that works for all cases sufficiently. Based on the previous experiences, I believe that it should be based on distances between positions and the cache should optimally keep entries with equal distances between the entries. In addition, there might be a need to simply increase the default snapshot cache size, snapshot timeout time and/or make it configurable with namespace level policies. btw. I've previously created a mermaidjs diagram to show the main interactions in replicated subscriptions: https://gist.github.com/lhotari/96fda511a70d7de93744d868b4472b92 That might be helpful to understand the flow. However, it seems that you have already found out how it works. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
