[jira] [Updated] (SPARK-18787) spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer usage by Netty
[ https://issues.apache.org/jira/browse/SPARK-18787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18787: - Labels: bulk-closed (was: ) > spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer > usage by Netty > --- > > Key: SPARK-18787 > URL: https://issues.apache.org/jira/browse/SPARK-18787 > Project: Spark > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Aniket Bhatnagar >Priority: Major > Labels: bulk-closed > > The documentation for the configuration spark.shuffle.io.preferDirectBufs > states that it will force all allocations from Netty to be on-heap but this > currently does not happen. The reason is that preferDirect parameter of > Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap > by Netty. In order to completely stop netty from using off heap memory, we > need to set the following system properties: > - io.netty.noUnsafe=true > - io.netty.threadLocalDirectBufferSize=0 > The proposal is to set properties (using System.setProperties) when the > executor starts (before any of the Netty classes load) or document these > properties to hint users on how to completely eliminate Netty' off heap > footprint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18787) spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer usage by Netty
[ https://issues.apache.org/jira/browse/SPARK-18787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Bhatnagar updated SPARK-18787: - Description: The documentation for the configuration spark.shuffle.io.preferDirectBufs states that it will force all allocations from Netty to be on-heap but this currently does not happen. The reason is that preferDirect parameter of Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty. In order to completely stop netty from using off heap memory, we need to set the following system properties: - io.netty.noUnsafe=true - io.netty.threadLocalDirectBufferSize=0 The proposal is to set properties (using System.setProperties) when the executor starts (before any of the Netty classes load) or document these properties to hint users on how to completely eliminate Netty' off heap footprint. was: The documentation for the configuration spark.shuffle.io.preferDirectBufs states that it will force all allocations from Netty to be on-heap but this currently does not happen. The reason is that preferDirect of Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty. In order to completely stop netty from using off heap memory, we need to set the following system properties: - io.netty.noUnsafe=true - io.netty.threadLocalDirectBufferSize=0 The proposal is to set properties (using System.setProperties) when the executor starts (before any of the Netty classes load) or document these properties to hint users on how to completely eliminate Netty' off heap footprint. > spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer > usage by Netty > --- > > Key: SPARK-18787 > URL: https://issues.apache.org/jira/browse/SPARK-18787 > Project: Spark > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Aniket Bhatnagar > > The documentation for the configuration spark.shuffle.io.preferDirectBufs > states that it will force all allocations from Netty to be on-heap but this > currently does not happen. The reason is that preferDirect parameter of > Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap > by Netty. In order to completely stop netty from using off heap memory, we > need to set the following system properties: > - io.netty.noUnsafe=true > - io.netty.threadLocalDirectBufferSize=0 > The proposal is to set properties (using System.setProperties) when the > executor starts (before any of the Netty classes load) or document these > properties to hint users on how to completely eliminate Netty' off heap > footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18787) spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer usage by Netty
[ https://issues.apache.org/jira/browse/SPARK-18787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Bhatnagar updated SPARK-18787: - Description: The documentation for the configuration spark.shuffle.io.preferDirectBufs states that it will force all allocations from Netty to be on-heap but this currently does not happen. The reason is that preferDirect of Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty. In order to completely stop netty from using off heap memory, we need to set the following system properties: - io.netty.noUnsafe=true - io.netty.threadLocalDirectBufferSize=0 The proposal is to set properties (using System.setProperties) when the executor starts (before any of the Netty classes load) or document these properties to hint users on how to completely eliminate Netty' off heap footprint. was: The documentation for the configuration spark.shuffle.io.preferDirectBufs states that it will force all allocations from Netty to be on-heap but this currently does not happen. The reason is that preferDirect of Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty. In order to completely stop netty from using off heap memory, we need to set the following system properties: - io.netty.noUnsafe=true - io.netty.threadLocalDirectBufferSize=0 The proposal is to set properties (using System.setProperties) when the executor starts (before any of the Netty classes load) or document these properties to hint users on how to completely eliminate off Netty' heap footprint. > spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer > usage by Netty > --- > > Key: SPARK-18787 > URL: https://issues.apache.org/jira/browse/SPARK-18787 > Project: Spark > Issue Type: Bug >Affects Versions: 2.0.2 >Reporter: Aniket Bhatnagar > > The documentation for the configuration spark.shuffle.io.preferDirectBufs > states that it will force all allocations from Netty to be on-heap but this > currently does not happen. The reason is that preferDirect of Netty's > PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty. > In order to completely stop netty from using off heap memory, we need to set > the following system properties: > - io.netty.noUnsafe=true > - io.netty.threadLocalDirectBufferSize=0 > The proposal is to set properties (using System.setProperties) when the > executor starts (before any of the Netty classes load) or document these > properties to hint users on how to completely eliminate Netty' off heap > footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org