[spark] branch master updated: [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers without regard for configuration

[srowen]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 055bf8e  [SPARK-24938][CORE] Prevent Netty from using onheap memory 
for headers without regard for configuration
055bf8e is described below

commit 055bf8ea1fe8dd4109772a282909ac3d69627f1b
Author: Nihar Sheth <niharrsh...@gmail.com>
AuthorDate: Tue Jan 22 08:41:42 2019 -0600

    [SPARK-24938][CORE] Prevent Netty from using onheap memory for headers 
without regard for configuration
    
    ## What changes were proposed in this pull request?
    
    In MessageEncoder.java, the header would always be allocated on onheap 
memory regardless of whether netty was configured to use/prefer onheap or 
offheap. By default this made netty allocate 16mb of onheap memory for a tiny 
header message. It would be more practical to use preallocated buffers.
    
    Using a memory monitor tool on a simple spark application, the following 
services currently allocate 16 mb of onheap memory:
    netty-rpc-client
    netty-blockTransfer-client
    netty-external-shuffle-client
    
    With this change, the memory monitor tool reports all three of these 
services as using 0 b of onheap memory. The offheap memory allocation does not 
increase, but more of the already-allocated space is used.
    
    ## How was this patch tested?
    
    Manually tested change using spark-memory-tool 
https://github.com/squito/spark-memory
    
    Closes #22114 from NiharS/nettybuffer.
    
    Lead-authored-by: Nihar Sheth <niharrsh...@gmail.com>
    Co-authored-by: Nihar Sheth <nsh...@cloudera.com>
    Signed-off-by: Sean Owen <sean.o...@databricks.com>
---
 .../src/main/java/org/apache/spark/network/protocol/MessageEncoder.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
 
b/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
index 997f74e..06dc447 100644
--- 
a/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
+++ 
b/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java
@@ -79,7 +79,7 @@ public final class MessageEncoder extends 
MessageToMessageEncoder<Message> {
     // sent.
     int headerLength = 8 + msgType.encodedLength() + in.encodedLength();
     long frameLength = headerLength + (isBodyInFrame ? bodyLength : 0);
-    ByteBuf header = ctx.alloc().heapBuffer(headerLength);
+    ByteBuf header = ctx.alloc().buffer(headerLength);
     header.writeLong(frameLength);
     msgType.encode(header);
     in.encode(header);


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org