[
https://issues.apache.org/jira/browse/HDFS-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739131#comment-13739131
]
Brandon Li commented on HDFS-4953:
----------------------------------
Some comments and questions:
1. DFSClient: looks like all the DFSClient instances share the same
ClientMmapManager instance. If this is the case, why not have one static
ClientMmapManager with a refcount to it, and remove ClientMmapManagerFactory
class and variable mmapManager?
2. HdfsZeroCopyCursor: might want to also initialize allowShortReads in the
constructor.
3. HdfsZeroCopyCursor: readViaSlowPath() throws when shortReads is disallowed
but fallbackBuffer is not provided. Since shortReads is false by default, the
user has to remember to setFallbackBuffer before doing zero copy read. Not sure
which case is more expected by the users, shortReads allowed or disallowed.
4. DFSInputStream: remove unused import and add debug level check for
DFSClient.LOG.Debug().
5. TestBlockReader: Assume.assumeTrue(SystemUtils.IS_OS_UNIX), guess you meant
IS_OS_LINUX
6. test_libhdfs_zerocopy.c: remove repeated
hdfsBuilderConfSetStr(bld, "dfs.block.size",
TO_STR(TEST_ZEROCOPY_FULL_BLOCK_SIZE));
7. TestBlockReaderLocal.java: remove unused import
8. please add javadoc to some classes, e.g., ClientMap,ClientMapManager
> enable HDFS local reads via mmap
> --------------------------------
>
> Key: HDFS-4953
> URL: https://issues.apache.org/jira/browse/HDFS-4953
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 2.3.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: benchmark.png, HDFS-4953.001.patch, HDFS-4953.002.patch,
> HDFS-4953.003.patch, HDFS-4953.004.patch, HDFS-4953.005.patch,
> HDFS-4953.006.patch
>
>
> Currently, the short-circuit local read pathway allows HDFS clients to access
> files directly without going through the DataNode. However, all of these
> reads involve a copy at the operating system level, since they rely on the
> read() / pread() / etc family of kernel interfaces.
> We would like to enable HDFS to read local files via mmap. This would enable
> truly zero-copy reads.
> In the initial implementation, zero-copy reads will only be performed when
> checksums were disabled. Later, we can use the DataNode's cache awareness to
> only perform zero-copy reads when we know that checksum has already been
> verified.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
