[jira] [Commented] (HDFS-1323) Pool/share file channels for HDFS read
[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044678#comment-13044678 ] Todd Lipcon commented on HDFS-1323: --- Some thoughts on the patch while I was updating it: - The ReadChannelPool class needs to have some updated javadoc explaining what exactly it does. - The synchronization isn't entirely clear within the class - looks like some things are both using the rwlock and concurrent collections, which always makes me suspicious - it probably makes more sense for this to be a least-recently-used cache rather than least-recently-created - not clear why we have a high-water and low-water mark, instead of just a single limit, where we close exactly one channel if we exceed the limit. - should be a conf to completely take the cache out of the loop > Pool/share file channels for HDFS read > -- > > Key: HDFS-1323 > URL: https://issues.apache.org/jira/browse/HDFS-1323 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Jay Booth > Attachments: hdfs-1323-20100730.patch, hdfs-1323-trunk.txt > > > Currently, all reads in HDFS require opening and closing the underlying > block/meta filechannels. We could pool these filechannels and save some > system calls and other work. Since HDFS read requests can be satisfied by > positioned reads and transferTos, we can even share these filechannels > between concurrently executing requests. > The attached patch was benchmarked as part of work on HDFS-918 and exhibited > a 10% performance increase for small random reads. > This does not affect client logic and involves minimal change to server > logic. Patch is based on branch 20-append. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1323) Pool/share file channels for HDFS read
[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044689#comment-13044689 ] Hadoop QA commented on HDFS-1323: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481528/hdfs-1323-trunk.txt against trunk revision 1131331. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.TestFileAppend2 +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/709//testReport/ Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/709//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/709//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/709//console This message is automatically generated. > Pool/share file channels for HDFS read > -- > > Key: HDFS-1323 > URL: https://issues.apache.org/jira/browse/HDFS-1323 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Jay Booth > Attachments: hdfs-1323-20100730.patch, hdfs-1323-trunk.txt > > > Currently, all reads in HDFS require opening and closing the underlying > block/meta filechannels. We could pool these filechannels and save some > system calls and other work. Since HDFS read requests can be satisfied by > positioned reads and transferTos, we can even share these filechannels > between concurrently executing requests. > The attached patch was benchmarked as part of work on HDFS-918 and exhibited > a 10% performance increase for small random reads. > This does not affect client logic and involves minimal change to server > logic. Patch is based on branch 20-append. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1323) Pool/share file channels for HDFS read
[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051066#comment-13051066 ] Jay Booth commented on HDFS-1323: - I could take a crack at it next week/weekend. Will get my build set up and report back in a week or two. > Pool/share file channels for HDFS read > -- > > Key: HDFS-1323 > URL: https://issues.apache.org/jira/browse/HDFS-1323 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Jay Booth > Attachments: hdfs-1323-20100730.patch, hdfs-1323-trunk.txt > > > Currently, all reads in HDFS require opening and closing the underlying > block/meta filechannels. We could pool these filechannels and save some > system calls and other work. Since HDFS read requests can be satisfied by > positioned reads and transferTos, we can even share these filechannels > between concurrently executing requests. > The attached patch was benchmarked as part of work on HDFS-918 and exhibited > a 10% performance increase for small random reads. > This does not affect client logic and involves minimal change to server > logic. Patch is based on branch 20-append. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1323) Pool/share file channels for HDFS read
[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894072#action_12894072 ] Jay Booth commented on HDFS-1323: - Correction - the patch created a 10% performance increase for HBase random GETs. It was probably a larger % of the read operation, if you don't include other work by HBase. > Pool/share file channels for HDFS read > -- > > Key: HDFS-1323 > URL: https://issues.apache.org/jira/browse/HDFS-1323 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Jay Booth > Fix For: 0.20-append, 0.22.0 > > Attachments: hdfs-1323-20100730.patch > > > Currently, all reads in HDFS require opening and closing the underlying > block/meta filechannels. We could pool these filechannels and save some > system calls and other work. Since HDFS read requests can be satisfied by > positioned reads and transferTos, we can even share these filechannels > between concurrently executing requests. > The attached patch was benchmarked as part of work on HDFS-918 and exhibited > a 10% performance increase for small random reads. > This does not affect client logic and involves minimal change to server > logic. Patch is based on branch 20-append. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
