Sital Kedia created SPARK-17839:
-----------------------------------
Summary: UnsafeSorterSpillReader should use Nio's directbuffer to
read the spill files in order to avoid additional copy
Key: SPARK-17839
URL: https://issues.apache.org/jira/browse/SPARK-17839
Project: Spark
Issue Type: Improvement
Components: Shuffle
Affects Versions: 2.0.1
Reporter: Sital Kedia
Priority: Minor
Currently we use BufferedInputStream to read the shuffle file which copies the
file content from os buffer cache to the user buffer. This adds additional
latency in reading the spill files. We made a change to use java nio's direct
buffer to read the spill files and for certain jobs spilling significant amount
of data, we see between 5 - 7% speedup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
