[
https://issues.apache.org/jira/browse/HDFS-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707587#comment-13707587
]
Todd Lipcon commented on HDFS-4953:
-----------------------------------
Oh, one other note about the benchmark graphs: you see some bimodal behavior in
some of the graphs where a portion of the reads run ~30% slower than the rest.
Our best assumption here is that this is NUMA behavior (the tests were run on a
dual-socket system). So, I think we should probably call mbind with
MPOL_INTERLEAVE on our mmaped segments so that we have more consistent
performance.
Another option would be to mbind the mapped segments to static nodes and then
expose the numa zone info to clients, so we can get socket-local scheduling.
But let's leave that for later ;-)
> enable HDFS local reads via mmap
> --------------------------------
>
> Key: HDFS-4953
> URL: https://issues.apache.org/jira/browse/HDFS-4953
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 2.2.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: benchmark.png, HDFS-4953.001.patch, HDFS-4953.002.patch
>
>
> Currently, the short-circuit local read pathway allows HDFS clients to access
> files directly without going through the DataNode. However, all of these
> reads involve a copy at the operating system level, since they rely on the
> read() / pread() / etc family of kernel interfaces.
> We would like to enable HDFS to read local files via mmap. This would enable
> truly zero-copy reads.
> In the initial implementation, zero-copy reads will only be performed when
> checksums were disabled. Later, we can use the DataNode's cache awareness to
> only perform zero-copy reads when we know that checksum has already been
> verified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
