[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549707#comment-13549707 ] Suresh Srinivas commented on HDFS-4353: --- Colin, can you add high level description of what you have changed in this patch? Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: _02a.patch, 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549959#comment-13549959 ] Colin Patrick McCabe commented on HDFS-4353: This patch refactors some code in the {{DFSClient}} and the DataNode's {{DataXceiver}}. The refactor encapsulates connections to peers into a single class named{{Peer}}. Suresh, please excuse me if I'm covering things you already know, but I want to give some context to random people reading this JIRA. Java has no standard mechanism for setting write timeouts on blocking sockets. So we usually wrap our sockets in {{org.apache.hadoop.net.SocketOutputStream}}. This class sets the {{Socket}} to nonblocking and simulates blocking I/O with a timeout. (There is also a parallel {{org.apache.hadoop.net.SocketInputStream}}.) However, we can't * always* do this, since some Sockets cannot be used in non-blocking mode-- for example, the SOCKS sockets classes don't support this. The other thing that we do a lot of is wrapping output and input streams in encrypted streams. The end result of this is that we end up passing around a lot of objects just to represent a single connection to a Peer. {{IOStreamPair}} is a good example of this. We also end up using {{instanceof}} a lot because we're dealing with types that don't have a common ancestor. This refactor encapsulates all of thos objects in a single object, the {{Peer}}. This avoids the need to use {{instanceof}} to set socket timeouts and other properties. The main reason for doing this refactor now is that {{DomainSocket}}, which is introduced by HDFS-4354, doesn't inherit from {{Socket}}. We made the decision not to inherit from {{Socket}} because inheriting would require us to rely on non-public JVM classes. There is more discsussion on HDFS-347 about this issue, if you're curious. Specific changes: {{PeerServer}}: a class that creates {{Peers}}. {{TcpPeerServer}} is basically a wrapper around {{ServerSocket}}. The next patch introduces another subclass, {{DomainPeerServer}}. {{BlockReader#close}}: now returns the Peer to the PeerCache directly. This replaces the multi-step process involving {{hasSentStatusCode}}, {{takeSocket}}, and{{getStreams}}. {{SocketCache}}: was renamed to {{PeerCache}}. Now caches based on {{DatanodeID}} rather than socket address. This is needed to prepare the way for puttingDomainSockets into the cache. Aside from that it should be very similar. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: _02a.patch, 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548370#comment-13548370 ] Hudson commented on HDFS-4353: -- Integrated in Hadoop-Yarn-trunk #91 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/91/]) svn merge -c -1430507 . for reverting HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes (Revision 1430662) HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes. Contributed by Colin Patrick McCabe. (Revision 1430507) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430662 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientBlockVerification.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDisableConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430507 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java *
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548477#comment-13548477 ] Hudson commented on HDFS-4353: -- Integrated in Hadoop-Hdfs-trunk #1280 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1280/]) svn merge -c -1430507 . for reverting HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes (Revision 1430662) HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes. Contributed by Colin Patrick McCabe. (Revision 1430507) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430662 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientBlockVerification.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDisableConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430507 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java *
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548493#comment-13548493 ] Hudson commented on HDFS-4353: -- Integrated in Hadoop-Mapreduce-trunk #1308 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1308/]) svn merge -c -1430507 . for reverting HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes (Revision 1430662) HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes. Contributed by Colin Patrick McCabe. (Revision 1430507) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430662 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientBlockVerification.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDisableConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430507 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java *
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549126#comment-13549126 ] Hadoop QA commented on HDFS-4353: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564027/_02a.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3811//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3811//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: _02a.patch, 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547249#comment-13547249 ] Hudson commented on HDFS-4353: -- Integrated in Hadoop-trunk-Commit #3194 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3194/]) HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes. Contributed by Colin Patrick McCabe. (Revision 1430507) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430507 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/BasicInetPeer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/EncryptedPeer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/NioInetPeer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/Peer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/PeerServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/TcpPeerServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientBlockVerification.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDisableConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0 Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch,
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547563#comment-13547563 ] Hudson commented on HDFS-4353: -- Integrated in Hadoop-trunk-Commit #3197 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3197/]) svn merge -c -1430507 . for reverting HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes (Revision 1430662) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430662 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientBlockVerification.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDisableConnCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPeerCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546217#comment-13546217 ] Todd Lipcon commented on HDFS-4353: --- Looking almost ready. A few minor nits: {code} + * it has been configured-- like when we're reading from the last replica + * of a block. {code} This description is a little unclear to me. Better to say such as when the caller has explicitly asked for a file to be opened without checksum verification. {code} public static BlockReader newBlockReader(Params params) throws IOException { +assert(params.getPeer() != null); +assert(params.getBlock() != null); +assert(params.getDatanodeID() != null); {code} I think {{Preconditions.checkArgument}} is more appropriate here - the checks are cheap (likely free) null checks, so we may as well always verify them. {code} + // TODO: create a wrapper class that makes channel-less sockets look like + // they have a channel, so that we can finally remove the legacy + // RemoteBlockReader. {code} Can you reference the JIRA for removing RBR here? I know we have one filed somewhere. +1 after these are addressed. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546246#comment-13546246 ] Todd Lipcon commented on HDFS-4353: --- Latest patch looks good. +1 pending Jenkins. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546476#comment-13546476 ] Hadoop QA commented on HDFS-4353: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563623/02f.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3787//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3787//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546524#comment-13546524 ] Todd Lipcon commented on HDFS-4353: --- I'll commit this later tonight or tomorrow morning unless there are any other comments. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch, 02f.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545652#comment-13545652 ] Colin Patrick McCabe commented on HDFS-4353: I just ran a few manual tests on this: running with the legacy blockreader, running with Kerberos configured, and running with {{dfs.encrypt.data.transfer}} set to true. They all worked. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544189#comment-13544189 ] Hadoop QA commented on HDFS-4353: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563247/02c.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3734//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544292#comment-13544292 ] Hadoop QA commented on HDFS-4353: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563368/02c.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:red}-1 javac{color:red}. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3737//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544511#comment-13544511 ] Hadoop QA commented on HDFS-4353: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563377/02d.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3738//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3738//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544568#comment-13544568 ] Hadoop QA commented on HDFS-4353: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563386/02e.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3740//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3740//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02c.patch, 02c.patch, 02-cumulative.patch, 02d.patch, 02e.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542872#comment-13542872 ] gschen commented on HDFS-4353: -- hi, how could I unsubscribe from this mailing list? thank you very much. justin Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543397#comment-13543397 ] Todd Lipcon commented on HDFS-4353: --- {code} -sock.setSoTimeout(dfsClient.getConf().socketTimeout); {code} did this timeout setting (just after the newPeer call) get lost in the refactor? {code} +private Peer peer; +private long time; {code} Make these final {code} + private Daemon daemon; {code} Put with the other variable declarations (below {{class Value}} - Rename {{scInstance}} to {{pcInstance}} or just {{instance}} - Javadoc for {{PeerCache.get(DataNode)}} is out of date - still refers to old params and types {code} + public synchronized void put(DatanodeID dnId, org.apache.hadoop.hdfs.net.Peer peer) { {code} Shouldn't need fully qualified class name here {code} -this.socketIn = NetUtils.getInputStream(s); -this.socketOut = NetUtils.getOutputStream(s, dnConf.socketWriteTimeout); -this.isLocal = s.getInetAddress().equals(s.getLocalAddress()); +this.socketIn = peer.getInputStream(); +this.socketOut = peer.getOutputStream(); {code} {{peer.getOutputStream}} here calls {{new SocketOutputStream(channel, 0)}} -- ie doesn't set a write timeout. However, the previous code here did set a write timeout. Does this turn out to not matter due to another call later? I have the same question in a number of other places where we used to pass a WRITE_TIMEOUT to {{NetUtils.getOutputStream}} but now appear to end up with timeout 0 (eg in {{newBlockReader}}). I see a few explicit calls to {{setWriteTimeout}} but can you please double check that we didn't miss any cases? Our test coverage of timeout behavior is light. {code} + * Free the resources associated with this peer factory. + * This normally includes sockets, etc. + * + * @throws IOException If there is an error closing the PeerFactory {code} Refers to factory when it should say server. The {{peerServer}} JavaDoc should explain a little bit more about what kind of 'tracking' it does of connections. I find it a little incongruous that {{PeerServer}} is responsible for tracking the set of connected peers. Since all connected clients have the same {{Peer}} interface, couldn't it stay as part of DataXceiverServer (ie just rename {{childSockets}} to {{childPeers}} or {{connectedPeers}}? Also seems strange to me that the {{PeerServer}} interface would have {{removePeer}} but not {{addPeer}}. Along the same lines, it seems odd that {{Peer.close}} would be responsible for removing itself from the {{peerServer}} - it's kind of inverted responsibility here, and odd since Peers are also used on clients with no associated server. -{{TcpPeerServer}} missing audience annotation Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543416#comment-13543416 ] Hadoop QA commented on HDFS-4353: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563135/02-cumulative.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 7 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.ha.TestActiveStandbyElectorRealZK org.apache.hadoop.ha.TestZKFailoverControllerStress org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3720//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3720//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3720//console This message is automatically generated. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543522#comment-13543522 ] Colin Patrick McCabe commented on HDFS-4353: bq. did this timeout setting (just after the newPeer call) get lost in the refactor? Originally: read timeout set in {{newSocket}}, write timeout set in {{BlockReaderFactory#newBlockReader}} Now: both timeouts set in {{BlockReaderFactory#newBlockReader}} bq. peer.getOutputStream here calls new SocketOutputStream(channel, 0) – ie doesn't set a write timeout. However, the previous code here did set a write timeout. Does this turn out to not matter due to another call later? {{DataXceiverServer#run}} sets a write timeout on the peer before passing it to {{DataXceiver}}. However, {{DataXceiver}} plays with the write timeout a lot. I suppose we could move the write timeout into {{DataXceiver}} too, just to keep them in the same place. [test failures] * Looks like I have to put the disable peer cache test into a separate junit test, due to our use of a singleton here. Also, {{LinkedListMultimap}} never calls {{equals}} at all, it seems, so I can leave off that part. Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543524#comment-13543524 ] Colin Patrick McCabe commented on HDFS-4353: er, that should read DataXceiver plays with the *read* timeout a lot Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4353) Encapsulate connections to peers in Peer and PeerServer classes
[ https://issues.apache.org/jira/browse/HDFS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543591#comment-13543591 ] Vinod Kumar Vavilapalli commented on HDFS-4353: --- Send an email to hdfs-dev-unsubscr...@hadoop.apache.org . Thanks, +Vinod Encapsulate connections to peers in Peer and PeerServer classes --- Key: HDFS-4353 URL: https://issues.apache.org/jira/browse/HDFS-4353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, hdfs-client Affects Versions: 2.0.3-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: 02b-cumulative.patch, 02-cumulative.patch Encapsulate connections to peers into the {{Peer}} and {{PeerServer}} classes. Since many Java classes may be involved with these connections, it makes sense to create a container for them. For example, a connection to a peer may have an input stream, output stream, readablebytechannel, encrypted output stream, and encrypted input stream associated with it. This makes us less dependent on the {{NetUtils}} methods which use {{instanceof}} to manipulate socket and stream states based on the runtime type. it also paves the way to introduce UNIX domain sockets which don't inherit from {{java.net.Socket}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira