Hi all, Thank you, Cong, for driving the 4.1.0 release process.
I'd like to propose considering the inclusion of PIP-430: Pulsar Broker Cache Improvements before cutting the branch-4.1. Here's my reasoning: Why PIP-430 would be valuable for 4.1: - Significant performance improvements: PIP-430 addresses fundamental inefficiencies in the current broker cache eviction mechanisms that impact CPU utilization, memory pressure, and cache hit rates - Better resource utilization: The current size-based eviction doesn't guarantee removal of the oldest entries globally, and timestamp-based eviction has high CPU overhead when iterating through all managed ledgers. This overhead is significant when the broker is serving a very large number of active topics. - Enhanced support for diverse workloads: The new "expected read count" strategy better handles mixed read patterns (tailing reads, catch-up reads, Key_Shared subscriptions) that are common in production Technical impact: - Reduces BookKeeper and tiered storage load through improved cache hit rates - Addresses performance issues with brokers handling large numbers of topics - Provides a foundation for future cache optimizations The PIP-430 implementation is very close to the finish line. The PR https://github.com/apache/pulsar/pull/24623 is waiting for reviews and after it has been merged, it would be ready for release. The first implementation PR of PIP-430 is already included in the master branch. In PIP-430, there are configuration options for choosing previous caching strategies if someone is concerned about regressions. However, extensive testing shows that the PIP-430 caching strategy performs better than other strategies in all use cases. Another PR that would need to be reviewed and merged before cutting branch-4.1 is the implementation PR for "PIP-428: Change TopicPoliciesService interface to fix consistency issues" at https://github.com/apache/pulsar/pull/24427. This addresses severe consistency issues in topic policy updates. There's yet another reason to wait for PIP-430 to be included: to have a valuable feature for Pulsar 4.1. There aren't very many actual changes between the master branch and branch-4.0 at the moment since most PIPs have also been backported to Pulsar 4.0.x. Many of the PIPs are minor improvements to existing features. Having a good reason for upgrading to Pulsar 4.1 would be useful. PIP-430 is such a feature. In summary: before we can proceed to cut branch-4.1, these PRs would need to be reviewed and merged: * https://github.com/apache/pulsar/pull/24623 (PIP-430 implementation, part 2) * https://github.com/apache/pulsar/pull/24427 (PIP-428 implementation) I'm looking forward to your reviews! -Lari On 2025/08/20 14:47:44 Cong Zhao wrote: > Hi community, > > It's been a long time since the 4.0 version was released. > > - 482 commits: > > https://github.com/apache/pulsar/pulls?q=is%3Apr+milestone%3A4.1.0+is%3Amerged+ > > > - 17 PIPs approved: > > https://github.com/apache/pulsar/pulls?q=is%3Apr+milestone%3A4.1.0+is%3Amerged++label%3APIP+ > > > We'd better prepare for the 4.1.0 release. I would like to cut branch-4.1 in > the next week. Please leave any ideas or concerns. > > Regards, > Cong Zhao >
