[ https://issues.apache.org/jira/browse/SPARK-21527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhoukang reopened SPARK-21527: ------------------------------ > Use buffer limit in order to take advantage of JAVA NIO Util's buffercache > --------------------------------------------------------------------------- > > Key: SPARK-21527 > URL: https://issues.apache.org/jira/browse/SPARK-21527 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.2.0 > Reporter: zhoukang > > Right now, ChunkedByteBuffer#writeFully do not slice bytes first.We observe > code in java nio Util below: > {code:java} > public static ByteBuffer More ...getTemporaryDirectBuffer(int size) { > BufferCache cache = bufferCache.get(); > ByteBuffer buf = cache.get(size); > if (buf != null) { > return buf; > } else { > // No suitable buffer in the cache so we need to allocate a new > // one. To avoid the cache growing then we remove the first > // buffer from the cache and free it. > if (!cache.isEmpty()) { > buf = cache.removeFirst(); > free(buf); > } > return ByteBuffer.allocateDirect(size); > } > } > {code} > If we slice first with a fixed size, we can use buffer cache and only need to > allocate at the first write call. > Since we allocate new buffer, we can not control the free time of this > buffer.This once cause memory issue in our production cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org