[
https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933597#comment-13933597
]
Chris Nauroth commented on MAPREDUCE-5791:
------------------------------------------
Hi, [~nikola.vujic]. This is an interesting find. Conventional wisdom is that
{{FileChannel#transferTo}} outperforms intermediate buffer copying like this.
bq. The reason is that transferTo method for the java.nio is issuing 32K IO
requests all the time.
Did you find more details behind this part? Is this because our code is
calling {{transferTo}} with the count parameter set to 32K, or is Windows
chopping a larger {{transferTo}} into multiple I/O requests for 32K each
internally?
Basically, I'm wondering if there is an alternative approach to force
{{transferTo}} to use the desired size on each I/O instead of reverting to
intermediate buffer copying.
> Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not
> read disks efficiently
> ------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5791
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Nikola Vujic
> Assignee: Nikola Vujic
> Attachments: MAPREDUCE-5791.patch
>
>
> transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using
> transferTo method from a FileChannel to transfer data from a disk to socket.
> This is performing slow in Windows, slower than in Linux. The reason is that
> transferTo method for the java.nio is issuing 32K IO requests all the time.
> In Windows, these 32K transfers are not optimal and we don't get the best
> performance form the underlying IO subsystem. In order to achieve better
> performance when reading from the drives, we need to read data in bigger
> chunks, 512K for example.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
