[ https://issues.apache.org/jira/browse/SPARK-18787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733902#comment-15733902 ]
Sean Owen commented on SPARK-18787: ----------------------------------- CC [~zsxwing] The tricky thing is that I think these classes may load before we know whether this config is enabled? > spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer > usage by Netty > ------------------------------------------------------------------------------------------- > > Key: SPARK-18787 > URL: https://issues.apache.org/jira/browse/SPARK-18787 > Project: Spark > Issue Type: Bug > Affects Versions: 2.0.2 > Reporter: Aniket Bhatnagar > > The documentation for the configuration spark.shuffle.io.preferDirectBufs > states that it will force all allocations from Netty to be on-heap but this > currently does not happen. The reason is that preferDirect of Netty's > PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty. > In order to completely stop netty from using off heap memory, we need to set > the following system properties: > - io.netty.noUnsafe=true > - io.netty.threadLocalDirectBufferSize=0 > The proposal is to set properties (using System.setProperties) when the > executor starts (before any of the Netty classes load) or document these > properties to hint users on how to completely eliminate off Netty' heap > footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org