Hi community, In our Celeborn cluster, the MALLOC_ARENA_MAX environment variable is not configured on the Worker machines. Without this configuration, glibc defaults to 8 * number of CPU cores. Since our machines have 128 cores, MALLOC_ARENA_MAX defaults to $8 \times 128 = 1024$.
Under this setup, the memory occupied by glibc arenas in the Worker process is extremely high. For example, arena=7.58GiB, heaps=545, and totalRSS=23.17GiB. Furthermore, the memory consumed by these arenas is not accounted for in Direct memory usage by the MemoryManager, leading to an excessively high overall memory footprint for the Worker process. Question： Our research indicates that many big data components recommend setting MALLOC_ARENA_MAX to 4 or 8. Does the Celeborn community have a recommended value for this? Additionally, are there any future plans to address the issue where Direct memory usage does not account for glibc arena memory?
