[jira] [Commented] (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464524#comment-13464524 ] LiuLei commented on HDFS-918: - Hi everyone, What is state of the issues, can I applied hdfs-91

[jira] [Commented] (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044677#comment-13044677 ] Todd Lipcon commented on HDFS-918: -- I rebased just the pooling part of the patch and posted

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978564#action_12978564 ] stack commented on HDFS-918: I was testing full patch. Seemed fine. (I like the sound of an eas

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977544#action_12977544 ] Jay Booth commented on HDFS-918: Hey all, sorry for the slow response, been swamped with the

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976984#action_12976984 ] stack commented on HDFS-918: Ran some read/write loading over a ten node cluster and all seems to

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976873#action_12976873 ] stack commented on HDFS-918: I'm game for enable/disable flag. I did some basic load testing ove

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976853#action_12976853 ] Todd Lipcon commented on HDFS-918: -- bq. When you say we could get it into 0.23, you thinking

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976804#action_12976804 ] stack commented on HDFS-918: I ran unit tests and the main difference was failure of '[junit] Te

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976347#action_12976347 ] Todd Lipcon commented on HDFS-918: -- Another thought: it should be trivial to enable/disable

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976346#action_12976346 ] Todd Lipcon commented on HDFS-918: -- Just took a swing through this patch. A few thoughts: -

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976340#action_12976340 ] stack commented on HDFS-918: I applied the patch on 0.20-append branch. It applies cleanly still

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976304#action_12976304 ] Edward Capriolo commented on HDFS-918: -- Fundamentally using selectors is more efficient

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893807#action_12893807 ] Jean-Daniel Cryans commented on HDFS-918: - I tested the latest full patch (not the po

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852832#action_12852832 ] Jay Booth commented on HDFS-918: Well gosh darn.. maybe I should just borrow your scripts an

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852682#action_12852682 ] Andrew Purtell commented on HDFS-918: - Sorry, test which contains HBase regionserver abor

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852681#action_12852681 ] Andrew Purtell commented on HDFS-918: - Same result as before with the patch "hdfs-918-bra

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852610#action_12852610 ] Jay Booth commented on HDFS-918: Am I ever, this one should be good to go but I'm on my way o

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852595#action_12852595 ] Andrew Purtell commented on HDFS-918: - Interested in another test with a 0.20 branch patc

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851869#action_12851869 ] Jay Booth commented on HDFS-918: Weird. We've been running an almost-the-same version of the

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851709#action_12851709 ] Andrew Purtell commented on HDFS-918: - I applied hdfs-918-branch20.2.patch to vanilla 0.2

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844943#action_12844943 ] Zlatin Balevsky commented on HDFS-918: -- The max and current sizes of the threadpool real

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844658#action_12844658 ] Jay Booth commented on HDFS-918: >>> I think it is very important to have separate pools for

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844620#action_12844620 ] Raghu Angadi commented on HDFS-918: --- > RE: Netty, I'm not very knowledgeable about it beyon

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844596#action_12844596 ] Jay Booth commented on HDFS-918: .bq I think it is very important to have separate pools for

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844485#action_12844485 ] Zlatin Balevsky commented on HDFS-918: -- bq. I think it is very important to have separat

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844280#action_12844280 ] Todd Lipcon commented on HDFS-918: -- bq. I think it is better to use something like Netty rat

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844278#action_12844278 ] Raghu Angadi commented on HDFS-918: --- Thanks Jay. This is a well known limitation on DataNo

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844208#action_12844208 ] Todd Lipcon commented on HDFS-918: -- +1 on ignoring write for now. This mainly benefits high

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844195#action_12844195 ] ryan rawson commented on HDFS-918: -- this sounds great guys. For HBase, there is more need fo

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844191#action_12844191 ] Jay Booth commented on HDFS-918: I'll enthusiastically cheerlead? :) More seriously, I'm wi

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844185#action_12844185 ] dhruba borthakur commented on HDFS-918: --- Thanks for the description Jay. What is ur fee

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844174#action_12844174 ] Jay Booth commented on HDFS-918: Yeah, it only uses nonblocking pread ops on the block and bl

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12844160#action_12844160 ] dhruba borthakur commented on HDFS-918: --- Thsi patch uses nio to multiplex IO over socke

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843559#action_12843559 ] Zlatin Balevsky commented on HDFS-918: -- bq. Current DFS : 92MB/s over 60 runs bq. Multi

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834321#action_12834321 ] Jay Booth commented on HDFS-918: Thanks Zlatin, I think you're right. I'll look at finding a

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834274#action_12834274 ] Zlatin Balevsky commented on HDFS-918: -- Jay, the selector thread is likely busylooping

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834252#action_12834252 ] Jay Booth commented on HDFS-918: Yeah, I'll do my best to get benchmarks by the end of the we

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834010#action_12834010 ] Todd Lipcon commented on HDFS-918: -- Have a few comments on the code, but before I write an i

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833579#action_12833579 ] Zlatin Balevsky commented on HDFS-918: -- I see a problem with doing the disk read on the

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831715#action_12831715 ] Jay Booth commented on HDFS-918: Absolutely -- every packet send has to fetch the BlockChanne

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831696#action_12831696 ] ryan rawson commented on HDFS-918: -- Great work, but Todd is dead on, we need to look at the

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831694#action_12831694 ] Todd Lipcon commented on HDFS-918: -- The 5M/sec vs 1.5M/sec for contended read seems pretty b

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831692#action_12831692 ] Jay Booth commented on HDFS-918: BENCHMARK UPDATE: For single threaded reads of 100MB, the n

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829368#action_12829368 ] ryan rawson commented on HDFS-918: -- the problem was we were using a stateful interface previ

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829364#action_12829364 ] dhruba borthakur commented on HDFS-918: --- > because we want to push as many parallel rea

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829293#action_12829293 ] ryan rawson commented on HDFS-918: -- I have done some thinking about HBase performance in rel

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829218#action_12829218 ] stack commented on HDFS-918: Yeah, I wouldn't rely on setPriority having an effect, at least on p

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828009#action_12828009 ] Jay Booth commented on HDFS-918: I haven't had a chance to run benchmarks yet, but I think th

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806647#action_12806647 ] dhruba borthakur commented on HDFS-918: --- The CPU time needed to dynamically create a th

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806476#action_12806476 ] Jay Booth commented on HDFS-918: Good call Todd, that's definitely the canonical case for mul

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806461#action_12806461 ] Todd Lipcon commented on HDFS-918: -- I think a main benefit of this work will be scalability

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806460#action_12806460 ] Jay Booth commented on HDFS-918: I'll definitely perform lots of benchmarking before asking f

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806419#action_12806419 ] dhruba borthakur commented on HDFS-918: --- This is good work. > it could make things si

[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads

[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804561#action_12804561 ] Jay Booth commented on HDFS-918: Current patch is having some issues in terms of actual use -