Hi all, In Pulsar, backlogs consist of unacknowledged messages. Once a message is acknowledged by all subscriptions, it enters the retention stage, as clearly illustrated in the official documentation [1]. The backlog quota limits the size and/or time threshold of the backlog, while the retention policy dictates the duration or volume of messages (acknowledged by all subscriptions) to retain.
However, I encountered an undocumented restriction that mandates the backlog quota to be greater than the retention policy. For example, a namespace with a backlog quota 20GB in size and unlimited in time cannot have a 2GB in size and 7 days in time retention policy. This detail is implemented in the code [3]. This restriction poses challenges in scenarios where there's a need for a larger backlog quota to accommodate extensive usage while desiring a shorter retention period for traceback purposes. Such configurations seem unattainable under the current constraint. Given that this limitation has been part of Pulsar since its initial open-source release, I am curious about the rationale behind it. Understanding the original intent or the technical considerations that led to this restriction would be immensely helpful. Therefore, I propose reevaluating and potentially removing this restriction to allow for more flexible configurations of backlog quota and retention policy. I would greatly appreciate your thoughts on this matter. Thank you for your time and consideration. I look forward to your response and am eager to contribute further to this discussion. [1]: https://pulsar.apache.org/docs/next/cookbooks-retention-expiry/#retention-policies [3]: https://github.com/apache/pulsar/blob/7315aeb6258b7adc9d874268d50acb95ffc0cf2b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java#L374-L393 Regards, Yike
