zhijiangW commented on issue #10180: [FLINK-14631] Account for netty direct allocations in direct memory limit (Netty Shuffle) URL: https://github.com/apache/flink/pull/10180#issuecomment-553760932 Thanks @azagrebin for this fixing. Some background information to share from my side: ATM the netty internal memory overhead should be small on output side. After the zero-copy improvement for input side working well via [7368](https://github.com/apache/flink/pull/7368), the whole netty memory overhead should be very small, even ignored. I wished to make this thing happen in next release, but not guarantee it. Based on the current situation, the netty memory overhead can reach even 1GB in large scale job sometimes. The number of arenas was coupled with the number of slots before by default, because this concept is a bit complex for users. Unless netty experts, it is difficult for users to understand how many memory would be allocated within one arena. Actually the number of arenas should be related to the number of physical connections and netty threads. Another option is to let users config this portion of memory size directly, which is somehow better understood, and the framework can make the transformation of arenas internally. My comment is not the blocker for this PR merging. :)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
