geniusjoe commented on issue #25021: URL: https://github.com/apache/pulsar/issues/25021#issuecomment-3615161785
> > Broker Operating system and hardware type: 5.4.241-1-tlinux4 (cloud vendor host Linux distributions, acted like CentOS) > > Since this is glibc based, have you considered tuning glibc parameters? [#24692](https://github.com/apache/pulsar/pull/24692) mentions some of the options. One other detail that is relevant in Linux is the Transparent Huge Pages configuration. It's necessary to tune the Linux defaults. Some details in another context (ZGC tuning) at [gunnarmorling/discussions.morling.dev#335 (comment)](https://github.com/gunnarmorling/discussions.morling.dev/discussions/335#discussioncomment-14441044) . > > For oom kills, one reason could be native memory heap fragmentation. Tuning the parameters could help in that. It's usually necessary to set at least `/sys/kernel/mm/transparent_hugepage/enable` to `madvise` or add `transparent_hugepage=madvise` boot parameter to get reasonable settings. In addition to that setting `MALLOC_ARENA_MAX=4` is something that reduces heap fragmentation with some tradeoffs. Many Java bases systems recommend setting `MALLOC_ARENA_MAX=4` on glibc. Thanks Lari. This is a good approach to optimize native memory recycling strategy. Since it involves modifications to Linux kernel parameters, I need to schedule time to implement and validate these changes. If the parameter adjustments improve significantly, I will provide feedback in this issue. ```Bash echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo advise | sudo tee /sys/kernel/mm/transparent_hugepage/shmem_enabled echo defer | sudo tee /sys/kernel/mm/transparent_hugepage/defrag echo 1 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag ``` The current Linux configuration on my broker is as follows: ```Bash # cat /sys/kernel/mm/transparent_hugepage/enabled always [madvise] never # cat /sys/kernel/mm/transparent_hugepage/shmem_enabled always within_size advise [never] deny force # cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise [madvise] never # cat /sys/kernel/mm/transparent_hugepage/khugepaged/defrag 1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
