[jira] [Commented] (HDFS-1323) Pool/share file channels for HDFS read

[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051066#comment-13051066 ] Jay Booth commented on HDFS-1323: - I could take a crack at it next week/weekend. Will get

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12977544#action_12977544 ] Jay Booth commented on HDFS-918: Hey all, sorry for the slow response, been swamped with the

[jira] Created: (HDFS-1323) Pool/share file channels for HDFS read

Pool/share file channels for HDFS read -- Key: HDFS-1323 URL: https://issues.apache.org/jira/browse/HDFS-1323 Project: Hadoop HDFS Issue Type: Improvement Components: data-node

[jira] Updated: (HDFS-1323) Pool/share file channels for HDFS read

[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-1323: Attachment: hdfs-1323-20100730.patch Pool/share file channels for HDFS read

[jira] Commented: (HDFS-1323) Pool/share file channels for HDFS read

[ https://issues.apache.org/jira/browse/HDFS-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12894072#action_12894072 ] Jay Booth commented on HDFS-1323: - Correction - the patch created a 10% performance increase

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hbase-hdfs-benchmarks.ods Benchmarked on EC2 this weekend, I set up 0.20.2-append clean, a copy

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-branch20-append.patch Managed to get back to this. Rebased on branch-20-append. Fixed

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852610#action_12852610 ] Jay Booth commented on HDFS-918: Am I ever, this one should be good to go but I'm on my way

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: (was: hdfs-918-branch20.2.patch) Use single Selector and small thread pool to replace many

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-branch20.2.patch Straightened out the block not found thing with Andrew, that was on his

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-TRUNK.patch Trunk patch with previous fixes. Use single Selector and small thread

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-branch20.2.patch Cleaned up a bug in the BlockChannelPool.cleanup() code, added new unit

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: (was: hdfs-200+826+918-branch20.patch) Use single Selector and small thread pool to replace

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-200+826+918-branch20.patch I heard all the cool kids are running HDFS-200 and HDFS-826 on

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: (was: hdfs-918-0.20.2.patch) Use single Selector and small thread pool to replace many

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-0.20.2.patch 0.20.2 compatible patch! A couple people mentioned that it would be much

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12844596#action_12844596 ] Jay Booth commented on HDFS-918: .bq I think it is very important to have separate pools for

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12844658#action_12844658 ] Jay Booth commented on HDFS-918: I think it is very important to have separate pools for

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12844174#action_12844174 ] Jay Booth commented on HDFS-918: Yeah, it only uses nonblocking pread ops on the block and

[jira] Commented: (HDFS-1034) Enhance datanode to read data and checksum file in parallel

[ https://issues.apache.org/jira/browse/HDFS-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12844186#action_12844186 ] Jay Booth commented on HDFS-1034: - Wouldn't this preclude the use of transferTo for transfer

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12844191#action_12844191 ] Jay Booth commented on HDFS-918: I'll enthusiastically cheerlead? :) More seriously, I'm

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-20100309.patch New patch and better benchmarks: Environment: 8x2GHz, 7GB RAM, namenode

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-20100228.patch New patch -- Took Zlatin's advice and utilized selectionKey.interestOps(0)

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12834252#action_12834252 ] Jay Booth commented on HDFS-918: Yeah, I'll do my best to get benchmarks by the end of the

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12834321#action_12834321 ] Jay Booth commented on HDFS-918: Thanks Zlatin, I think you're right. I'll look at finding a

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-20100203.patch New patch. Streamlined MultiplexedBlockSender, we now have one selector

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-918-20100201.patch New patch.. * new configuration params:

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12828009#action_12828009 ] Jay Booth commented on HDFS-918: I haven't had a chance to run benchmarks yet, but I think

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806460#action_12806460 ] Jay Booth commented on HDFS-918: I'll definitely perform lots of benchmarking before asking

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806476#action_12806476 ] Jay Booth commented on HDFS-918: Good call Todd, that's definitely the canonical case for

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12804561#action_12804561 ] Jay Booth commented on HDFS-918: Current patch is having some issues in terms of actual use

[jira] Created: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

Use single Selector and small thread pool to replace many instances of BlockSender for reads Key: HDFS-918 URL: https://issues.apache.org/jira/browse/HDFS-918

[jira] Updated: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-918: --- Attachment: hdfs-multiplex.patch Here's a first implementation -- it works, passes TestDistributedFileSystem,

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) This work didn't really go anywhere, I

[jira] Commented: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12755823#action_12755823 ] Jay Booth commented on HDFS-516: Yeah, I was puzzled by the performance too. I dug through

[jira] Commented: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12755120#action_12755120 ] Jay Booth commented on HDFS-516: Hey Todd, in short, I agree, we should be looking at moving

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: hdfs-516-20090831.patch) Low Latency distributed reads -

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) Low Latency distributed reads -

[jira] Commented: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12750246#action_12750246 ] Jay Booth commented on HDFS-516: I did some benchmarking, here are the results: Each test

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: hdfs-516-20090831.patch New patch, IPC server was too slow for IO operations (like 40 times slower

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: hdfs-516-20090824.patch Changed caching system to ehcache, which gives us a configurable api with

[jira] Commented: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12738651#action_12738651 ] Jay Booth commented on HDFS-516: Wow, thanks Raghu, that's awesome and will save me a ton of

[jira] Created: (HDFS-516) Low Latency distributed reads

Low Latency distributed reads - Key: HDFS-516 URL: https://issues.apache.org/jira/browse/HDFS-516 Project: Hadoop HDFS Issue Type: New Feature Reporter: Jay Booth Priority: Minor I

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Status: Patch Available (was: Open) Here's the initial patch. I have some pretty decent byte-consistency and

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.patch Low Latency distributed reads - Key:

[jira] Commented: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737713#action_12737713 ] Jay Booth commented on HDFS-516: Here's some architectural overview and a general request for

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.tgz Oops. Shows what I get for posting at the end of the day, svn diff misses new files,

[jira] Commented: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737799#action_12737799 ] Jay Booth commented on HDFS-516: Ok, thanks, is one big patch preferred to the tiny patch +

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) Low Latency distributed reads -

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) Low Latency distributed reads -

[jira] Updated: (HDFS-516) Low Latency distributed reads

[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.patch Low Latency distributed reads - Key: