[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201812#comment-14201812
]
Aaron Davidson commented on SPARK-2468:
---------------------------------------
Looking at the netty code a bit more, it seems that they might unconditionally
allocate direct buffers for IO, whether or not direct is "preferred".
Additionally, they allocate more memory based on the number of cores in your
system. The default settings would be roughly 16MB per core, and this might be
multiplied by 2 in our current setup since we have independent client and
server pools in the same JVM. I'm not certain how executors running in YARN
report "availableProcessors", but is it possible your machines have 32 or
greater cores? This could cause an extra allocation of around 1GB direct heap
buffers.
> Netty-based block server / client module
> ----------------------------------------
>
> Key: SPARK-2468
> URL: https://issues.apache.org/jira/browse/SPARK-2468
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Reporter: Reynold Xin
> Assignee: Reynold Xin
> Priority: Critical
> Fix For: 1.2.0
>
>
> Right now shuffle send goes through the block manager. This is inefficient
> because it requires loading a block from disk into a kernel buffer, then into
> a user space buffer, and then back to a kernel send buffer before it reaches
> the NIC. It does multiple copies of the data and context switching between
> kernel/user. It also creates unnecessary buffer in the JVM that increases GC
> Instead, we should use FileChannel.transferTo, which handles this in the
> kernel space with zero-copy. See
> http://www.ibm.com/developerworks/library/j-zerocopy/
> One potential solution is to use Netty. Spark already has a Netty based
> network module implemented (org.apache.spark.network.netty). However, it
> lacks some functionality and is turned off by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
