[ https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803974#action_12803974 ]
Todd Lipcon commented on HDFS-916: ---------------------------------- Yes, I'd probably want to use Netty. I haven't given it enough research yet to decide, but I read up a bit on it earlier this week and it seems pretty reasonable. I don't want to start a religious war between different NIO frameworks here, though ;-) Let's first answer: are people interested in seeing and reviewing a patch that is essentially a rewrite of DFSOutputStream? > Rewrite DFSOutputStream to use a single thread with NIO > ------------------------------------------------------- > > Key: HDFS-916 > URL: https://issues.apache.org/jira/browse/HDFS-916 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client > Affects Versions: 0.22.0 > Reporter: Todd Lipcon > Assignee: Todd Lipcon > > The DFS write pipeline code has some really hairy multi-threaded > synchronization. There have been a lot of bugs produced by this (HDFS-101, > HDFS-793, HDFS-915, tens of others) since it's very hard to understand the > message passing, lock sharing, and interruption properties. The reason for > the multiple threads is to be able to simultaneously send and receive. If > instead of using multiple threads, it used nonblocking IO, I think the whole > thing would be a lot less error prone. > I think we could do this in two halves: one half is the DFSOutputStream. The > other half is BlockReceiver. I opened this JIRA first as I think it's simpler > (only one TCP connection to deal with, rather than an up and downstream) > Opinions? Am I crazy? I would like to see some agreement on the idea before I > spend time writing code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.