[
https://issues.apache.org/jira/browse/SPARK-17839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560381#comment-15560381
]
Apache Spark commented on SPARK-17839:
--------------------------------------
User 'sitalkedia' has created a pull request for this issue:
https://github.com/apache/spark/pull/15408
> UnsafeSorterSpillReader should use Nio's directbuffer to read the spill files
> in order to avoid additional copy
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17839
> URL: https://issues.apache.org/jira/browse/SPARK-17839
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.0.1
> Reporter: Sital Kedia
> Priority: Minor
>
> Currently we use BufferedInputStream to read the shuffle file which copies
> the file content from os buffer cache to the user buffer. This adds
> additional latency in reading the spill files. We made a change to use java
> nio's direct buffer to read the spill files and for certain jobs spilling
> significant amount of data, we see between 5 - 7% speedup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
