Here's a hint as to why this restriction was introduced.
After reviewing the current implementation of the retention policy, I found
that the logic for determining whether a ledger exceeds the retention size and
should be deleted is based on the following logic (pseudocode):
```
long totalSizeToDelete = 0;
for (ledger in ledgersBeforeSlowestReaderLedgerId) {
totalSizeToDelete += ledger.size;
if (managedLedgerTotalSize - totalSizeToDelete >= retentionSize) {
// add this ledger to ledgersToDelete list
}
}
```
The actual code can be seen at: [1]
The issue lies in the use of `managedLedgerTotalSize - totalSizeToDelete >=
retentionSize`, where `managedLedgerTotalSize` is used instead of
`SUM(ledgerSize(ledgersBeforeSlowestReaderLedgerId))`. This causes ledgers that
have not been acknowledged to also be included in the retention size comparison.
Here is an example to illustrate the problem:
Suppose the current ManagedLedger has 5 ledgers, [1, 2, 3, 4, 5], each ledger's
size is 1MB, and the retentionSize is set to 2MB. The current
slowestReaderLedgerId is 3. Based on the current implementation, ledgers 1 and
2 are expected to be retained but will be deleted.
The retention policy will only actually take effect when the retentionSize
exceeds the backlog quota.
[1]:
https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2695-L2696
Regards,
Yike
________________________________
From: Yike Xiao <km...@live.com>
Sent: Wednesday, April 3, 2024 15:36
To: dev@pulsar.apache.org <dev@pulsar.apache.org>
Subject: [DISCUSS] Remove the limitation between backlog quota and retention
policy
Hi all,
In Pulsar, backlogs consist of unacknowledged messages. Once a message is
acknowledged by all subscriptions, it enters the retention stage, as clearly
illustrated in the official documentation [1]. The backlog quota limits the
size and/or time threshold of the backlog, while the retention policy dictates
the duration or volume of messages (acknowledged by all subscriptions) to
retain.
However, I encountered an undocumented restriction that mandates the backlog
quota to be greater than the retention policy. For example, a namespace with a
backlog quota 20GB in size and unlimited in time cannot have a 2GB in size and
7 days in time retention policy. This detail is implemented in the code [3].
This restriction poses challenges in scenarios where there's a need for a
larger backlog quota to accommodate extensive usage while desiring a shorter
retention period for traceback purposes. Such configurations seem unattainable
under the current constraint.
Given that this limitation has been part of Pulsar since its initial
open-source release, I am curious about the rationale behind it. Understanding
the original intent or the technical considerations that led to this
restriction would be immensely helpful.
Therefore, I propose reevaluating and potentially removing this restriction to
allow for more flexible configurations of backlog quota and retention policy.
I would greatly appreciate your thoughts on this matter.
Thank you for your time and consideration. I look forward to your response and
am eager to contribute further to this discussion.
[1]:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpulsar.apache.org%2Fdocs%2Fnext%2Fcookbooks-retention-expiry%2F%23retention-policies&data=05%7C02%7C%7Ccb059c912d5a4f98327308dc53b0ded0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638477266343295533%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=wRjCYXkwGq4TAhYet8FdxkpnaehpCay7ST1AZi2jKAo%3D&reserved=0<https://pulsar.apache.org/docs/next/cookbooks-retention-expiry/#retention-policies>
[3]:
https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar%2Fblob%2F7315aeb6258b7adc9d874268d50acb95ffc0cf2b%2Fpulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2FAdminResource.java%23L374-L393&data=05%7C02%7C%7Ccb059c912d5a4f98327308dc53b0ded0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638477266343313621%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=JmKZUCH%2F9dD07pGIknGpOnTfGUNmaDAn3bM7f%2FirgKg%3D&reserved=0<https://github.com/apache/pulsar/blob/7315aeb6258b7adc9d874268d50acb95ffc0cf2b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java#L374-L393>
Regards,
Yike
