lhotari commented on issue #25021: URL: https://github.com/apache/pulsar/issues/25021#issuecomment-3584774110
There a blog post by AutoMQ folks about PooledByteBufAllocator memory fragmentation: https://www.automq.com/blog/netty-based-streaming-systems-memory-fragmentation-and-oom-issues . I guess the blog post was written before [AdaptiveByteBufAllocator](https://netty.io/4.1/api/io/netty/buffer/AdaptiveByteBufAllocator.html) became usable. It would be interesting to also see how AdaptiveByteBufAllocator behaves in Pulsar. In Pulsar, the broker caching adds more of this fragmentation since it could hold on to much larger buffer when an entry is cached. In Netty, a slice of a buffer will hold the parent buffer in memory until all slices have been released. The mitigation in Pulsar is `managedLedgerCacheCopyEntries=true`, but that adds overhead since entries would get copied each time they are added to the cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
