[ https://issues.apache.org/jira/browse/HDFS-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer resolved HDFS-281. ----------------------------------- Resolution: Won't Fix I'm going to close this as a duplicate of HDFS-2246. While the path chosen was different, the end result was essentially the same. > Explore usage of the sendfile api via > java.nio.channels.FileChannel.transfer{To|From} for i/o in datanodes > ---------------------------------------------------------------------------------------------------------- > > Key: HDFS-281 > URL: https://issues.apache.org/jira/browse/HDFS-281 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Arun C Murthy > > We could potentially gain a lot of performance by using the *sendfile* system > call: > $ man sendfile > {noformat} > DESCRIPTION > This call copies data between one file descriptor and another. > Either or both of these file descriptors may refer to a socket (but see > below). > in_fd should be a file descriptor opened for reading and out_fd should > be a descriptor opened for writing. offset is a pointer to a variable > holding the input file pointer position from which sendfile() will > start reading data. When sendfile() returns, this variable will be set to the > offset of the byte following the last byte that was read. count is > the number of bytes to copy between file descriptors. > Because this copying is done within the kernel, sendfile() does not > need to spend time transferring data to and from user space. > {noformat} > The nio package offers this via the > java.nio.channels.FileChannel.transfer{To|From} apis: > http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/FileChannel.html#transferFrom(java.nio.channels.ReadableByteChannel,%20long,%20long) > http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/FileChannel.html#transferTo(long,%20long,%20java.nio.channels.WritableByteChannel) > From the javadocs: > {noformat} > This method is potentially much more efficient than a simple loop that > reads from this channel and writes to the target channel. Many operating > systems can transfer bytes directly from the filesystem cache to the target > channel without actually copying them. > {noformat} > ---- > Hence, this could well-worth exploring for doing io at the datanodes... -- This message was sent by Atlassian JIRA (v6.2#6252)