[jira] [Updated] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2653: -- Resolution: Fixed Fix Version/s: 1.1.0 0.23.1 Target Version/s: (was: 1.1.0, 0.23.1) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Todd. I've committed this. DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.23.1, 1.1.0 Attachments: hdfs-2653-1.patch, hdfs-2653-2.patch, hdfs-2653-2.patch, hdfs-2653-b1-1.patch, hdfs-2653-b1-2.patch Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168202#comment-13168202 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Hdfs-trunk-Commit #1481 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1481/]) HDFS-2653. DFSClient should cache whether addrs are non-local when short-circuiting is enabled. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213586 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.23.1, 1.1.0 Attachments: hdfs-2653-1.patch, hdfs-2653-2.patch, hdfs-2653-2.patch, hdfs-2653-b1-1.patch, hdfs-2653-b1-2.patch Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168205#comment-13168205 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Common-trunk-Commit #1407 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1407/]) HDFS-2653. DFSClient should cache whether addrs are non-local when short-circuiting is enabled. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213586 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.23.1, 1.1.0 Attachments: hdfs-2653-1.patch, hdfs-2653-2.patch, hdfs-2653-2.patch, hdfs-2653-b1-1.patch, hdfs-2653-b1-2.patch Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168206#comment-13168206 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Hdfs-0.23-Commit #265 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/265/]) HDFS-2653. svn merge -c 1213586 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213589 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0
[jira] [Updated] (HDFS-2661) Enable protobuf RPC for DatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2661: --- Status: Patch Available (was: Open) Enable protobuf RPC for DatanodeProtocol Key: HDFS-2661 URL: https://issues.apache.org/jira/browse/HDFS-2661 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch This jira changes the client and servers to use protobuf based RPC for DatanodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2661) Enable protobuf RPC for DatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2661: --- Attachment: HDFS-2661.trunk.patch Enable protobuf RPC for DatanodeProtocol Key: HDFS-2661 URL: https://issues.apache.org/jira/browse/HDFS-2661 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch This jira changes the client and servers to use protobuf based RPC for DatanodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2661) Enable protobuf RPC for DatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2661: --- Status: Open (was: Patch Available) Enable protobuf RPC for DatanodeProtocol Key: HDFS-2661 URL: https://issues.apache.org/jira/browse/HDFS-2661 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch This jira changes the client and servers to use protobuf based RPC for DatanodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168213#comment-13168213 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Hdfs-trunk-Commit #1482 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1482/]) HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213592 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168215#comment-13168215 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Common-trunk-Commit #1408 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1408/]) HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213592 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2654: -- Attachment: hdfs-2654-3.patch Wrt trunk feedback, took your suggestions and committed. Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168219#comment-13168219 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Hdfs-0.23-Commit #266 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/266/]) HDFS-2654. svn merge -c 1213592/1213593 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213594 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168222#comment-13168222 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Common-0.23-Commit #276 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/276/]) HDFS-2654. svn merge -c 1213592/1213593 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213594 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168227#comment-13168227 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Mapreduce-0.23-Commit #287 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/287/]) HDFS-2653. svn merge -c 1213586 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213589 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1,
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168228#comment-13168228 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1431 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1431/]) HDFS-2653. DFSClient should cache whether addrs are non-local when short-circuiting is enabled. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213586 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.23.1, 1.1.0 Attachments: hdfs-2653-1.patch, hdfs-2653-2.patch, hdfs-2653-2.patch, hdfs-2653-b1-1.patch, hdfs-2653-b1-2.patch Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168238#comment-13168238 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1432 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1432/]) HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213592 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2654: -- Attachment: hdfs-2654-b1-4.patch bq. prefer to rename it to RemoteBlockReader instead of BlockReaderRemote Done. bq. what's the reason to add this new closeStream method in addition to close(), instead of just making the BlockReader interface extend Closeable? That's better. Made BlockReader extend Closeable and removed closeStream (by having BlockReader#close call IOUtils#closeStream on both streams instead of using IOUtils#closeStream(reader) to do this. Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168239#comment-13168239 ] Eli Collins commented on HDFS-2654: --- Updated branch-1 patch attached. Look good? Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168245#comment-13168245 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Mapreduce-0.23-Commit #288 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/288/]) HDFS-2654. svn merge -c 1213592/1213593 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213594 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Make BlockReaderLocal not extend RemoteBlockReader2 --- Key:
[jira] [Updated] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-2669: --- Attachment: pbEnableClientNN4.patch Enable protobuf rpc for ClientNamenodeProtocol -- Key: HDFS-2669 URL: https://issues.apache.org/jira/browse/HDFS-2669 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: pbEnableClientNN1.patch, pbEnableClientNN4.patch Change the client and servers to use protobuf based RPC ClientNamenodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-2669: --- Status: Patch Available (was: Open) Enable protobuf rpc for ClientNamenodeProtocol -- Key: HDFS-2669 URL: https://issues.apache.org/jira/browse/HDFS-2669 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: pbEnableClientNN1.patch, pbEnableClientNN4.patch Change the client and servers to use protobuf based RPC ClientNamenodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2661) Enable protobuf RPC for DatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168293#comment-13168293 ] Hadoop QA commented on HDFS-2661: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507144/HDFS-2661.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1702//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1702//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1702//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1702//console This message is automatically generated. Enable protobuf RPC for DatanodeProtocol Key: HDFS-2661 URL: https://issues.apache.org/jira/browse/HDFS-2661 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch, HDFS-2661.trunk.patch This jira changes the client and servers to use protobuf based RPC for DatanodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2672) possible Cases for NullPointerException
possible Cases for NullPointerException
---
Key: HDFS-2672
URL: https://issues.apache.org/jira/browse/HDFS-2672
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: kavita sharma
Priority: Trivial
NullPointerException can be thrown as null check is not added in DFSClient and
UpgradeManagerNamenode.
{noformat}
DFSClient.java
Block newBlock = primary.getBlockInfo(last.getBlock());
if newBlock comes null then NullPointerException will be thrown.
{noformat}
{noformat}
UpgradeManagerNamenode.java
uos = UpgradeObjectCollection.getDistributedUpgrades(-4,
HdfsConstants.NodeType.NAME_NODE);
if uos will come null then NullPointerException will be thrown.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168329#comment-13168329 ] Hadoop QA commented on HDFS-2669: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507152/pbEnableClientNN4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1703//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1703//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1703//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1703//console This message is automatically generated. Enable protobuf rpc for ClientNamenodeProtocol -- Key: HDFS-2669 URL: https://issues.apache.org/jira/browse/HDFS-2669 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: pbEnableClientNN1.patch, pbEnableClientNN4.patch Change the client and servers to use protobuf based RPC ClientNamenodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168344#comment-13168344 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Hdfs-0.23-Build #106 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/106/]) HDFS-2653. svn merge -c 1213586 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213589 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168346#comment-13168346 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Hdfs-0.23-Build #106 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/106/]) HDFS-2654. svn merge -c 1213592/1213593 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213594 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168365#comment-13168365 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Mapreduce-0.23-Build #124 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/124/]) HDFS-2654. svn merge -c 1213592/1213593 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213594 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Make BlockReaderLocal not extend RemoteBlockReader2 --- Key:
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168363#comment-13168363 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Mapreduce-0.23-Build #124 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/124/]) HDFS-2653. svn merge -c 1213586 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213589 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project/src/site DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0
[jira] [Commented] (HDFS-1765) Block Replication should respect under-replication block priority
[ https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168426#comment-13168426 ] Hudson commented on HDFS-1765: -- Integrated in Hadoop-Hdfs-trunk #893 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/893/]) Put HDFS-1765 in the right place in the changelog. HDFS-1765. Block Replication should respect under-replication block priority. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213537 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java Block Replication should respect under-replication block priority - Key: HDFS-1765 URL: https://issues.apache.org/jira/browse/HDFS-1765 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Hairong Kuang Assignee: Uma Maheswara Rao G Fix For: 0.24.0 Attachments: HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.pdf, underReplicatedQueue.pdf Time Spent: 0.5h Remaining Estimate: 0h Currently under-replicated blocks are assigned different priorities depending on how many replicas a block has. However the replication monitor works on blocks in a round-robin fashion. So the newly added high priority blocks won't get replicated until all low-priority blocks are done. One example is that on decommissioning datanode WebUI we often observe that blocks with only decommissioning replicas do not get scheduled to replicate before other blocks, so risking data availability if the node is shutdown for repair before decommission completes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168429#comment-13168429 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Hdfs-trunk #893 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/893/]) HDFS-2663. Handle protobuf optional parameters correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213512 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDfsOverAvroRpc.java Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168434#comment-13168434 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Hdfs-trunk #893 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/893/]) HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213592 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168433#comment-13168433 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Hdfs-trunk #893 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/893/]) HDFS-2653. DFSClient should cache whether addrs are non-local when short-circuiting is enabled. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213586 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.23.1, 1.1.0 Attachments: hdfs-2653-1.patch, hdfs-2653-2.patch, hdfs-2653-2.patch, hdfs-2653-b1-1.patch, hdfs-2653-b1-2.patch Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1765) Block Replication should respect under-replication block priority
[ https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168442#comment-13168442 ] Hudson commented on HDFS-1765: -- Integrated in Hadoop-Mapreduce-trunk #926 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/926/]) Put HDFS-1765 in the right place in the changelog. HDFS-1765. Block Replication should respect under-replication block priority. Contributed by Uma Maheswara Rao G eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213537 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java Block Replication should respect under-replication block priority - Key: HDFS-1765 URL: https://issues.apache.org/jira/browse/HDFS-1765 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Hairong Kuang Assignee: Uma Maheswara Rao G Fix For: 0.24.0 Attachments: HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.pdf, underReplicatedQueue.pdf Time Spent: 0.5h Remaining Estimate: 0h Currently under-replicated blocks are assigned different priorities depending on how many replicas a block has. However the replication monitor works on blocks in a round-robin fashion. So the newly added high priority blocks won't get replicated until all low-priority blocks are done. One example is that on decommissioning datanode WebUI we often observe that blocks with only decommissioning replicas do not get scheduled to replicate before other blocks, so risking data availability if the node is shutdown for repair before decommission completes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2653) DFSClient should cache whether addrs are non-local when short-circuiting is enabled
[ https://issues.apache.org/jira/browse/HDFS-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168449#comment-13168449 ] Hudson commented on HDFS-2653: -- Integrated in Hadoop-Mapreduce-trunk #926 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/926/]) HDFS-2653. DFSClient should cache whether addrs are non-local when short-circuiting is enabled. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213586 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java DFSClient should cache whether addrs are non-local when short-circuiting is enabled --- Key: HDFS-2653 URL: https://issues.apache.org/jira/browse/HDFS-2653 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.23.1, 1.1.0 Attachments: hdfs-2653-1.patch, hdfs-2653-2.patch, hdfs-2653-2.patch, hdfs-2653-b1-1.patch, hdfs-2653-b1-2.patch Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168450#comment-13168450 ] Hudson commented on HDFS-2654: -- Integrated in Hadoop-Mapreduce-trunk #926 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/926/]) HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213592 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168445#comment-13168445 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Mapreduce-trunk #926 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/926/]) HDFS-2663. Handle protobuf optional parameters correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213512 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDfsOverAvroRpc.java Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2649) eclipse:eclipse build fails for hadoop-hdfs-httpfs
[ https://issues.apache.org/jira/browse/HDFS-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168490#comment-13168490 ] Jason Lowe commented on HDFS-2649: -- Thanks for the review, Alejandro! The javadoc, findbugs, and core test failures are unrelated to this patch. eclipse:eclipse build fails for hadoop-hdfs-httpfs -- Key: HDFS-2649 URL: https://issues.apache.org/jira/browse/HDFS-2649 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.24.0, 0.23.1 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: hdfs-2649.patch Building the eclipse:eclipse target fails in the hadoop-hdfs-httpfs project with this error: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-eclipse-plugin:2.8:eclipse (default-cli) on project hadoop-hdfs-httpfs: Request to merge when 'filtering' is not identical. Original=resource src/main/resources: output=target/classes, include=[httpfs.properties], exclude=[**/*.java], test=false, filtering=true, merging with=resource src/main/resources: output=target/classes, include=[], exclude=[httpfs.properties|**/*.java], test=false, filtering=false - [Help 1] This appears to be the same type of issue as reported in HADOOP-7567. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2649) eclipse:eclipse build fails for hadoop-hdfs-httpfs
[ https://issues.apache.org/jira/browse/HDFS-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168512#comment-13168512 ] Tom White commented on HDFS-2649: - +1 This fixes 'mvn eclipse:eclipse' for me, and the projects load correctly into Eclipse. eclipse:eclipse build fails for hadoop-hdfs-httpfs -- Key: HDFS-2649 URL: https://issues.apache.org/jira/browse/HDFS-2649 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.24.0, 0.23.1 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: hdfs-2649.patch Building the eclipse:eclipse target fails in the hadoop-hdfs-httpfs project with this error: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-eclipse-plugin:2.8:eclipse (default-cli) on project hadoop-hdfs-httpfs: Request to merge when 'filtering' is not identical. Original=resource src/main/resources: output=target/classes, include=[httpfs.properties], exclude=[**/*.java], test=false, filtering=true, merging with=resource src/main/resources: output=target/classes, include=[], exclude=[httpfs.properties|**/*.java], test=false, filtering=false - [Help 1] This appears to be the same type of issue as reported in HADOOP-7567. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2666) TestBackupNode fails
[ https://issues.apache.org/jira/browse/HDFS-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2666: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review Eli. I committed the patch. TestBackupNode fails Key: HDFS-2666 URL: https://issues.apache.org/jira/browse/HDFS-2666 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2666.txt TestBackupNode fails post the commit of HDFS-2647. For some reason, the jenkins does not flag this as test failure and silently ignores it! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-234: Attachment: HDFS-234.diff Moved to contrib. Added README for running a namenode with BookKeeper. Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2666) TestBackupNode fails
[ https://issues.apache.org/jira/browse/HDFS-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168551#comment-13168551 ] Hudson commented on HDFS-2666: -- Integrated in Hadoop-Common-trunk-Commit #1412 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1412/]) HDFS-2666. Fix TestBackupNode failure. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213808 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java TestBackupNode fails Key: HDFS-2666 URL: https://issues.apache.org/jira/browse/HDFS-2666 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2666.txt TestBackupNode fails post the commit of HDFS-2647. For some reason, the jenkins does not flag this as test failure and silently ignores it! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2650) Replace @inheritDoc with @Override
[ https://issues.apache.org/jira/browse/HDFS-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2650: -- Resolution: Fixed Status: Resolved (was: Patch Available) +1 for the patch. I committed the patch to trunk. Thank you Hari. Replace @inheritDoc with @Override --- Key: HDFS-2650 URL: https://issues.apache.org/jira/browse/HDFS-2650 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 0.24.0 Attachments: HDFS-2650-2.txt, HDFS-2650.txt, hdfs-2650 @Override provides both javadoc from superclass and compile time detection of overridden method deletion from superclass. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2673) While Namenode processing the blocksBeingWrittenReport, it will log incorrect number blocks count
While Namenode processing the blocksBeingWrittenReport, it will log incorrect
number blocks count
-
Key: HDFS-2673
URL: https://issues.apache.org/jira/browse/HDFS-2673
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Trivial
In NameNode#blocksBeingWrittenReport
we have the following stateChangeLog
{code}
stateChangeLog.info(*BLOCK* NameNode.blocksBeingWrittenReport:
+from +nodeReg.getName()+ +blocks.length + blocks);
{code}
here blocks is long array. Every consecutive 3 elements represents a block (
length, blockid, genstamp).
So, here in log message, blocks.length should be blocks.length/3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2666) TestBackupNode fails
[ https://issues.apache.org/jira/browse/HDFS-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168560#comment-13168560 ] Hudson commented on HDFS-2666: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1436 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1436/]) HDFS-2666. Fix TestBackupNode failure. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213808 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java TestBackupNode fails Key: HDFS-2666 URL: https://issues.apache.org/jira/browse/HDFS-2666 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2666.txt TestBackupNode fails post the commit of HDFS-2647. For some reason, the jenkins does not flag this as test failure and silently ignores it! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2650) Replace @inheritDoc with @Override
[ https://issues.apache.org/jira/browse/HDFS-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168561#comment-13168561 ] Hudson commented on HDFS-2650: -- Integrated in Hadoop-Hdfs-trunk-Commit #1487 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1487/]) HDFS-2650. Replace @inheritDoc with @Override. Contributed by Hari Mankude. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213813 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/CorruptFileBlockIterator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileChecksumServlets.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeRegistrationWritable.java Replace @inheritDoc with @Override --- Key: HDFS-2650 URL: https://issues.apache.org/jira/browse/HDFS-2650 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 0.24.0 Attachments: HDFS-2650-2.txt, HDFS-2650.txt, hdfs-2650 @Override provides both javadoc from superclass and compile time detection of overridden method deletion from superclass. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2650) Replace @inheritDoc with @Override
[ https://issues.apache.org/jira/browse/HDFS-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168562#comment-13168562 ] Hudson commented on HDFS-2650: -- Integrated in Hadoop-Common-trunk-Commit #1413 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1413/]) HDFS-2650. Replace @inheritDoc with @Override. Contributed by Hari Mankude. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213813 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/CorruptFileBlockIterator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileChecksumServlets.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeRegistrationWritable.java Replace @inheritDoc with @Override --- Key: HDFS-2650 URL: https://issues.apache.org/jira/browse/HDFS-2650 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 0.24.0 Attachments: HDFS-2650-2.txt, HDFS-2650.txt, hdfs-2650 @Override provides both javadoc from superclass and compile time detection of overridden method deletion from superclass. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2545) Webhdfs: Support multiple namenodes in federation
[ https://issues.apache.org/jira/browse/HDFS-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2545: -- Attachment: HDFS-2545.txt Given my comments were trivial, uploading a patch with changes I had suggested. Webhdfs: Support multiple namenodes in federation - Key: HDFS-2545 URL: https://issues.apache.org/jira/browse/HDFS-2545 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: HDFS-2545.txt, h2545_20111202.patch, h2545_20111205.patch DatanodeWebHdfsMethods only talks to the default namenode. It won't work if there are multiple namenodes in federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2545) Webhdfs: Support multiple namenodes in federation
[ https://issues.apache.org/jira/browse/HDFS-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168569#comment-13168569 ] Suresh Srinivas commented on HDFS-2545: --- Comments: # Some lines are more than 80 chars # Domain#toString() could be static method Other than the ptach looks good. +1. Webhdfs: Support multiple namenodes in federation - Key: HDFS-2545 URL: https://issues.apache.org/jira/browse/HDFS-2545 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: HDFS-2545.txt, h2545_20111202.patch, h2545_20111205.patch DatanodeWebHdfsMethods only talks to the default namenode. It won't work if there are multiple namenodes in federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168573#comment-13168573 ] Doug Cutting commented on HDFS-2660: Before protobuf conversion, the RPC server had protocol and corresponding implementation was implementation of that interface. With protobuf, the server has been changed to use ProtocolPB interface. That sounds like a change that violates the design of the RPCEngine API, which was meant to provide a serialization and transport-independent abstraction for RPC that the rest of Hadoop builds on. Does RPC engine pluggability mean, just change the RPC engine to some other engine and every thing continues work fine? That was the intent. If that's no longer desired, then RPCEngine should be removed and protobuf should be used directly. HDFS-2647 and now HDFS-2669 seem to be assuming that all RPC in HDFS will use protobuf and only protobuf, since they're directly specifying protobuf-specifics in e.g., NameNodeRpcServer.java. That class used to be generic, indepdendent of any particular RPC serialization or transport. Is there no way to constrain protobuf-specifics to ProtobufRpcEngine, as has been done for other RPC engines? If not, why can't we at least condition protobuf-specifics to cases where protobuf is actually being used, as in the patch that I provided. In other words, why is that patch insufficient? Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2647) Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, JournalProtocol and NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168581#comment-13168581 ] Doug Cutting commented on HDFS-2647: This should not have been committed without addressing the fact that it breaks TestDfsOverAvroRpc. Prior to this, NameNodeRpcServer was independent of the RPCEngine being used. One could switch the configured RPCEngine and HDFS would still work. After this HDFS seems to only work with the ProtobufRPCEngine. That's not simply 'enabling' protobuf RPC. That's mandating protobuf RPC. If that's desired then support for other RPC systems should probably be removed, e.g., remove TestDfsOverAvroRpc, remove AvroRPCEngine, remove Avro annotations on HDFS protocols, perhaps even remove the RPCEngine abstraction, since it would no longer perform any useful function. Either that, or figure out a way to make protobuf RPC not mandated, but rather merely the default. Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, JournalProtocol and NamenodeProtocol --- Key: HDFS-2647 URL: https://issues.apache.org/jira/browse/HDFS-2647 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, data-node, hdfs client, name-node Affects Versions: 0.24.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-2647.txt, HDFS-2647.txt This jira changes the client and servers to use protobuf based RPC for the protocols mentioned in the Summary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168582#comment-13168582 ] Doug Cutting commented on HDFS-2669: This furthers the breakage of TestDFSOverAvroRpc initiated by HDFS-2647. Please resolve that before compounding it. Enable protobuf rpc for ClientNamenodeProtocol -- Key: HDFS-2669 URL: https://issues.apache.org/jira/browse/HDFS-2669 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: pbEnableClientNN1.patch, pbEnableClientNN4.patch Change the client and servers to use protobuf based RPC ClientNamenodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2674) hdfs script does not work out of the box.
hdfs script does not work out of the box.
-
Key: HDFS-2674
URL: https://issues.apache.org/jira/browse/HDFS-2674
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Priority: Critical
Fix For: 0.24.0
As the title says, hadoop-config.sh doesn't add the hadoop-common jars, which
makes the hdfs script fail.
To repro, follow the instructions from
http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment
{code}
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:24 [0 jobs] [hist
1889]
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:29 [0 jobs] [hist
1890]
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:36 [0 jobs] [hist
1891]
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:42 [0 jobs] [hist
1892]
$ cat $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml EOF
?xml version=1.0?!-- core-site.xml --
configuration
property
namefs.default.name/name
valuehdfs://localhost//value
/property
/configuration
EOF
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:51 [0 jobs] [hist
1893]
$ hdfs namenode -format
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/HadoopIllegalArgumentException
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.HadoopIllegalArgumentException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2674) hdfs script does not work out of the box.
[
https://issues.apache.org/jira/browse/HDFS-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kelly updated HDFS-2674:
-
Attachment: HDFS-2674.diff
Fix is simple, hdfs-config.sh was setting HADOOP_PREFIX, while it shouldn't be.
Looks like this came in with HADOOP-7801. Before this, the setting of
HADOOP_PREFIX had no effect.
hdfs script does not work out of the box.
-
Key: HDFS-2674
URL: https://issues.apache.org/jira/browse/HDFS-2674
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Priority: Critical
Fix For: 0.24.0
Attachments: HDFS-2674.diff
As the title says, hadoop-config.sh doesn't add the hadoop-common jars, which
makes the hdfs script fail.
To repro, follow the instructions from
http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment
{code}
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:24 [0 jobs] [hist
1889]
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:29 [0 jobs] [hist
1890]
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:36 [0 jobs] [hist
1891]
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:42 [0 jobs] [hist
1892]
$ cat $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml EOF
?xml version=1.0?!-- core-site.xml --
configuration
property
namefs.default.name/name
valuehdfs://localhost//value
/property
/configuration
EOF
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:51 [0 jobs] [hist
1893]
$ hdfs namenode -format
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/HadoopIllegalArgumentException
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.HadoopIllegalArgumentException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2674) hdfs script does not work out of the box.
[
https://issues.apache.org/jira/browse/HDFS-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kelly updated HDFS-2674:
-
Status: Patch Available (was: Open)
hdfs script does not work out of the box.
-
Key: HDFS-2674
URL: https://issues.apache.org/jira/browse/HDFS-2674
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Priority: Critical
Fix For: 0.24.0
Attachments: HDFS-2674.diff
As the title says, hadoop-config.sh doesn't add the hadoop-common jars, which
makes the hdfs script fail.
To repro, follow the instructions from
http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment
{code}
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:24 [0 jobs] [hist
1889]
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:29 [0 jobs] [hist
1890]
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:36 [0 jobs] [hist
1891]
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:42 [0 jobs] [hist
1892]
$ cat $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml EOF
?xml version=1.0?!-- core-site.xml --
configuration
property
namefs.default.name/name
valuehdfs://localhost//value
/property
/configuration
EOF
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:51 [0 jobs] [hist
1893]
$ hdfs namenode -format
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/HadoopIllegalArgumentException
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.HadoopIllegalArgumentException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2650) Replace @inheritDoc with @Override
[ https://issues.apache.org/jira/browse/HDFS-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168601#comment-13168601 ] Hudson commented on HDFS-2650: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1437 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1437/]) HDFS-2650. Replace @inheritDoc with @Override. Contributed by Hari Mankude. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213813 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/CorruptFileBlockIterator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileChecksumServlets.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeRegistrationWritable.java Replace @inheritDoc with @Override --- Key: HDFS-2650 URL: https://issues.apache.org/jira/browse/HDFS-2650 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 0.24.0 Attachments: HDFS-2650-2.txt, HDFS-2650.txt, hdfs-2650 @Override provides both javadoc from superclass and compile time detection of overridden method deletion from superclass. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168603#comment-13168603 ] Doug Cutting commented on HDFS-2660: In order to support Avro, the right way to do it is, change the getServer() calls and addProtocol() calls to use WireProtocol and no longer just setting the RPC engine. What do you mean by WireProtocol here? The protobuf-generated Java interface? That contains, e.g., fields that are not meant to be transmitted but are not declared transient. In other words, it's not a very good specification of a wire protocol. Avro actually includes code that knows how to serialize protobuf structs but doesn't yet include protobuf-specific code for protocols (http://s.apache.org/MZT). That could be added so that protobuf becomes the 'serialization-independent' Hadoop protocol specification instead of using Java interfaces as we have before. I'd welcome that contribution to Avro. Then RPCEngine might then be modified to accept protobuf ServiceDescriptors to specify protocols rather than Java interfaces. Or we could abandon serialization-independent RPC as an architectural goal and remove the code related to it. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2670) Exclude javadoc generation for generated protobuf classes
[ https://issues.apache.org/jira/browse/HDFS-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168602#comment-13168602 ] Aaron T. Myers commented on HDFS-2670: -- +1, this would be very helpful. Generating javadocs for the protobuf code also takes a very long time. Exclude javadoc generation for generated protobuf classes - Key: HDFS-2670 URL: https://issues.apache.org/jira/browse/HDFS-2670 Project: Hadoop HDFS Issue Type: Improvement Components: build Affects Versions: 0.24.0 Reporter: Todd Lipcon Priority: Critical On my machine generating javadoc now uses multiple GB of disk space, most of it going to the HDFS protobuf API code. We should explicitly exclude these files from the javadoc build step. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1972) HA: Datanode fencing mechanism
[
https://issues.apache.org/jira/browse/HDFS-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168612#comment-13168612
]
Todd Lipcon commented on HDFS-1972:
---
Dhruba, that's an interesting idea. I hadn't considered whether
{{processMisReplicatedBlocks}} could be run in a background thread.
Do you do this by chunking up the work so you only have to hold the lock for
short periods of time? Or you think it can run entirely lockless?
I'll take a look at your github link today.
HA: Datanode fencing mechanism
--
Key: HDFS-1972
URL: https://issues.apache.org/jira/browse/HDFS-1972
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, ha, name-node
Reporter: Suresh Srinivas
Assignee: Todd Lipcon
Attachments: hdfs-1972-v1.txt, hdfs-1972.txt
In high availability setup, with an active and standby namenode, there is a
possibility of two namenodes sending commands to the datanode. The datanode
must honor commands from only the active namenode and reject the commands
from standby, to prevent corruption. This invariant must be complied with
during fail over and other states such as split brain. This jira addresses
issues related to this, design of the solution and implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168613#comment-13168613 ] Todd Lipcon commented on HDFS-2654: --- +1, branch-1 v4 looks good to me. Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1972) HA: Datanode fencing mechanism
[ https://issues.apache.org/jira/browse/HDFS-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168624#comment-13168624 ] dhruba borthakur commented on HDFS-1972: findOverReplicatedReplicas runs with the readLock. chooseExcessReplicates runs without any lock but returns it results rather than modifying any global data structures. Then we acquire the fsnamesystem lock, quickly validate that the state of the block has not changed since we did all the computation (and most of the time the state does not change), and then proceed to do the action. HA: Datanode fencing mechanism -- Key: HDFS-1972 URL: https://issues.apache.org/jira/browse/HDFS-1972 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, ha, name-node Reporter: Suresh Srinivas Assignee: Todd Lipcon Attachments: hdfs-1972-v1.txt, hdfs-1972.txt In high availability setup, with an active and standby namenode, there is a possibility of two namenodes sending commands to the datanode. The datanode must honor commands from only the active namenode and reject the commands from standby, to prevent corruption. This invariant must be complied with during fail over and other states such as split brain. This jira addresses issues related to this, design of the solution and implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2664) Remove TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168625#comment-13168625 ] Suresh Srinivas commented on HDFS-2664: --- Currently there is a lot of work that was done to cleanup pluggability of RPC engines. So I do not think any thing is broken in terms of pluggability of RPC engines. The issue is merely with the test, which expects to work by just changing the RPC engine for a protocol, without regard to how RPC implementation is done. The way AvroRpc tests are implemented currently, it depends on just replacing the RPC engine to AvroRpcEngine. This worked in the past only because, both writable and Avro use reflection to invoke server implementation. That is no longer the case with protobuf. I see couple of choices: 1. Instead of using HDFS for testing the pluggability, which has resulted in TestDfsOverAvroRpc being broken several times, I suggest writing a test in common, along the lines of TestProtoBufRpc. There are several writable rpc tests as well, along similar lines. This will help in not having to fix TestDfsOverAvroRpc as HDFS changes. 2. Change TestDfsOverAvroRpc to supply Writable based implementation of Namenode RPCs. This will fix the test. However my vote is to do the option 1. Remove TestDFSOverAvroRpc - Key: HDFS-2664 URL: https://issues.apache.org/jira/browse/HDFS-2664 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 With HDFS-2647, HDFS has transitioned to protocol buffers. The server side implementation registers ProtocolPB.class and a BlockingService as implementation. Client side uses ProtocolPB.class as the interface. The RPC engine used is protobuf both for the RPC proxy and the server. With this TestDFSOverAvroRpc fails. I propose removing this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2664) Remove TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168626#comment-13168626 ] Suresh Srinivas commented on HDFS-2664: --- Doug, I am seeing that there are several jiras where you are making comments related to this topic. Can we limit it to this jira and resolve it, to keep the discussion in one place. Remove TestDFSOverAvroRpc - Key: HDFS-2664 URL: https://issues.apache.org/jira/browse/HDFS-2664 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 With HDFS-2647, HDFS has transitioned to protocol buffers. The server side implementation registers ProtocolPB.class and a BlockingService as implementation. Client side uses ProtocolPB.class as the interface. The RPC engine used is protobuf both for the RPC proxy and the server. With this TestDFSOverAvroRpc fails. I propose removing this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2673) While Namenode processing the blocksBeingWrittenReport, it will log incorrect number blocks count
[
https://issues.apache.org/jira/browse/HDFS-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2673:
--
Attachment: HDFS-2673.patch
Updated patch for 1.1.0 version
While Namenode processing the blocksBeingWrittenReport, it will log incorrect
number blocks count
-
Key: HDFS-2673
URL: https://issues.apache.org/jira/browse/HDFS-2673
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Trivial
Attachments: HDFS-2673.patch
In NameNode#blocksBeingWrittenReport
we have the following stateChangeLog
{code}
stateChangeLog.info(*BLOCK* NameNode.blocksBeingWrittenReport:
+from +nodeReg.getName()+ +blocks.length + blocks);
{code}
here blocks is long array. Every consecutive 3 elements represents a block (
length, blockid, genstamp).
So, here in log message, blocks.length should be blocks.length/3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread
[
https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168657#comment-13168657
]
Jitendra Nath Pandey commented on HDFS-2589:
+1.
unnecessary hftp token fetch and renewal thread
---
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 1.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-2589.patch
Instantiation of the hftp filesystem is causing a token to be implicitly
created and added to a custom token renewal thread. With the new token
renewal feature in the JT, this causes the mapreduce
{{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled
token, and an explicit token) and leave a spurious renewal thread running.
This thread should not be running in the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and
to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2545) Webhdfs: Support multiple namenodes in federation
[ https://issues.apache.org/jira/browse/HDFS-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168682#comment-13168682 ] Hadoop QA commented on HDFS-2545: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507217/HDFS-2545.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.http.server.TestHttpFSServer org.apache.hadoop.lib.servlet.TestServerWebApp +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1705//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1705//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1705//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1705//console This message is automatically generated. Webhdfs: Support multiple namenodes in federation - Key: HDFS-2545 URL: https://issues.apache.org/jira/browse/HDFS-2545 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: HDFS-2545.txt, h2545_20111202.patch, h2545_20111205.patch DatanodeWebHdfsMethods only talks to the default namenode. It won't work if there are multiple namenodes in federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168691#comment-13168691 ] Hadoop QA commented on HDFS-234: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12507213/HDFS-234.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 90 warning messages. -1 javac. The applied patch generated 39 javac compiler warnings (more than the trunk's current 37 warnings). -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDFSRemove +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1704//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1704//console This message is automatically generated. Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2617) Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
[ https://issues.apache.org/jira/browse/HDFS-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168748#comment-13168748 ] Jakob Homan commented on HDFS-2617: --- @Aaron - think about this some more, and not hearing any comments, I think it'd be better to go with SPNEGO for a couple of reasons: (1) keep a consistent approach to the web interfaces for the NN/2NN (we could re-use tokens from the map-output fetch, but it would be a bit messy) and (2) the current kerbssl approach is used to fetch/renew/etc delegation tokens explicitly so we don't have to have an API call (to enabled hftp). Moving to SPNEGO for these would preserve this behavior. The next question is - how to deprecate the kerbssl. It'll be quite annoying to have to support both for a couple releases. Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution -- Key: HDFS-2617 URL: https://issues.apache.org/jira/browse/HDFS-2617 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jakob Homan Assignee: Jakob Homan The current approach to secure and authenticate nn web services is based on Kerberized SSL and was developed when a SPNEGO solution wasn't available. Now that we have one, we can get rid of the non-standard KSSL and use SPNEGO throughout. This will simplify setup and configuration. Also, Kerberized SSL is a non-standard approach with its own quirks and dark corners (HDFS-2386). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2654) Make BlockReaderLocal not extend RemoteBlockReader2
[ https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-2654. --- Resolution: Fixed Target Version/s: 0.23.1, 1.1.0 (was: 1.1.0, 0.23.1) Hadoop Flags: Reviewed Thanks Todd. Merged to branch-1. Make BlockReaderLocal not extend RemoteBlockReader2 --- Key: HDFS-2654 URL: https://issues.apache.org/jira/browse/HDFS-2654 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.1, 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2654-1.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-2.patch, hdfs-2654-3.patch, hdfs-2654-b1-1.patch, hdfs-2654-b1-2.patch, hdfs-2654-b1-3.patch, hdfs-2654-b1-4.patch The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share sifnificant code we can make the BlockReader interface an abstract class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2617) Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
[ https://issues.apache.org/jira/browse/HDFS-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168761#comment-13168761 ] Aaron T. Myers commented on HDFS-2617: -- @Jakob - those are certainly good reasons. Another benefit of going with the SPNEGO approach is that fsck just goes straight to a URL on the NN, so the approach I proposed would require adding some system to authenticate that as well. So, +1 to SPNEGO. It seems like my proposal would be more trouble than it's worth. Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution -- Key: HDFS-2617 URL: https://issues.apache.org/jira/browse/HDFS-2617 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jakob Homan Assignee: Jakob Homan The current approach to secure and authenticate nn web services is based on Kerberized SSL and was developed when a SPNEGO solution wasn't available. Now that we have one, we can get rid of the non-standard KSSL and use SPNEGO throughout. This will simplify setup and configuration. Also, Kerberized SSL is a non-standard approach with its own quirks and dark corners (HDFS-2386). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168764#comment-13168764 ] Jitendra Nath Pandey commented on HDFS-234: --- +1 for the patch. Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2675) Reduce verbosity when double-closing edit logs
Reduce verbosity when double-closing edit logs -- Key: HDFS-2675 URL: https://issues.apache.org/jira/browse/HDFS-2675 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial Currently the edit logs log at WARN level when they're double-closed. But this happens in the normal flow of things, so we may as well reduce it to DEBUG to reduce log spam in unit tests, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168788#comment-13168788 ] Suresh Srinivas commented on HDFS-2660: --- bq. That sounds like a change that violates the design of the RPCEngine API, which was meant to provide a serialization and transport-independent abstraction for RPC that the rest of Hadoop builds on. I am not sure what design is violated. Can you point me to javadoc related to this. Also these APIs are not marked stable. They are evolving. bq. Does RPC engine pluggability mean, just change the RPC engine to some other engine and every thing continues work fine? bq. That was the intent. If that's no longer desired, then RPCEngine should be removed and protobuf should be used directly. HDFS-2647 and now HDFS-2669 seem to be assuming that all RPC in HDFS will use protobuf and only protobuf, since they're directly specifying protobuf-specifics in e.g., NameNodeRpcServer.java. That class used to be generic, indepdendent of any particular RPC serialization or transport. To RPC we now pass three things: RpcKind, Interface, Implementation. The protobuf to which HDFS has switched to provides Implementation that does not share same java interface as the Interface. One could get Avro working by just passing suitable implementation for which reflection is useful. bq. Is there no way to constrain protobuf-specifics to ProtobufRpcEngine, as has been done for other RPC engines? If not, why can't we at least condition protobuf-specifics to cases where protobuf is actually being used, as in the patch that I provided. In other words, why is that patch insufficient? If you look at the results for your patch, you will see lot more tests failed. As I said earlier, the right implementation needs to be passed. bq. What do you mean by WireProtocol here? The protobuf-generated Java interface? That contains, e.g., fields that are not meant to be transmitted but are not declared transient. In other words, it's not a very good specification of a wire protocol. I meant ClientNodeWireProtocol, etc. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1972) HA: Datanode fencing mechanism
[ https://issues.apache.org/jira/browse/HDFS-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168791#comment-13168791 ] Suresh Srinivas commented on HDFS-1972: --- Oops sorry, wrong jira. HA: Datanode fencing mechanism -- Key: HDFS-1972 URL: https://issues.apache.org/jira/browse/HDFS-1972 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, ha, name-node Reporter: Suresh Srinivas Assignee: Todd Lipcon Attachments: hdfs-1972-v1.txt, hdfs-1972.txt In high availability setup, with an active and standby namenode, there is a possibility of two namenodes sending commands to the datanode. The datanode must honor commands from only the active namenode and reject the commands from standby, to prevent corruption. This invariant must be complied with during fail over and other states such as split brain. This jira addresses issues related to this, design of the solution and implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1972) HA: Datanode fencing mechanism
[ https://issues.apache.org/jira/browse/HDFS-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168790#comment-13168790 ] Suresh Srinivas commented on HDFS-1972: --- BTW again we have many discussions related to the same topic happening on diff jiras. It would be good we stick to HDFS-2664. HA: Datanode fencing mechanism -- Key: HDFS-1972 URL: https://issues.apache.org/jira/browse/HDFS-1972 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, ha, name-node Reporter: Suresh Srinivas Assignee: Todd Lipcon Attachments: hdfs-1972-v1.txt, hdfs-1972.txt In high availability setup, with an active and standby namenode, there is a possibility of two namenodes sending commands to the datanode. The datanode must honor commands from only the active namenode and reject the commands from standby, to prevent corruption. This invariant must be complied with during fail over and other states such as split brain. This jira addresses issues related to this, design of the solution and implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168792#comment-13168792 ] Suresh Srinivas commented on HDFS-2660: --- BTW again we have many discussions related to the same topic happening on diff jiras. It would be good we stick to HDFS-2664. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2617) Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
[ https://issues.apache.org/jira/browse/HDFS-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168800#comment-13168800 ] Owen O'Malley commented on HDFS-2617: - I'm +1 to SPNEGO too. Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution -- Key: HDFS-2617 URL: https://issues.apache.org/jira/browse/HDFS-2617 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jakob Homan Assignee: Jakob Homan The current approach to secure and authenticate nn web services is based on Kerberized SSL and was developed when a SPNEGO solution wasn't available. Now that we have one, we can get rid of the non-standard KSSL and use SPNEGO throughout. This will simplify setup and configuration. Also, Kerberized SSL is a non-standard approach with its own quirks and dark corners (HDFS-2386). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2674) hdfs script does not work out of the box.
[
https://issues.apache.org/jira/browse/HDFS-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168801#comment-13168801
]
Hadoop QA commented on HDFS-2674:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507221/HDFS-2674.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 javadoc. The javadoc tool appears to have generated 90 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 13 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1706//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/1706//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/1706//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1706//console
This message is automatically generated.
hdfs script does not work out of the box.
-
Key: HDFS-2674
URL: https://issues.apache.org/jira/browse/HDFS-2674
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Priority: Critical
Fix For: 0.24.0
Attachments: HDFS-2674.diff
As the title says, hadoop-config.sh doesn't add the hadoop-common jars, which
makes the hdfs script fail.
To repro, follow the instructions from
http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment
{code}
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:24 [0 jobs] [hist
1889]
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:29 [0 jobs] [hist
1890]
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:36 [0 jobs] [hist
1891]
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:42 [0 jobs] [hist
1892]
$ cat $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml EOF
?xml version=1.0?!-- core-site.xml --
configuration
property
namefs.default.name/name
valuehdfs://localhost//value
/property
/configuration
EOF
ivank@spokegrown-lm ~/src/hadoop-common Tue Dec 13 19:14:51 [0 jobs] [hist
1893]
$ hdfs namenode -format
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/HadoopIllegalArgumentException
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.HadoopIllegalArgumentException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168809#comment-13168809 ] Doug Cutting commented on HDFS-2660: One could get Avro working by just passing suitable implementation for which reflection is useful. Then please make this change or remove the Avro code. Please don't just disable tests. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2664) Remove TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168811#comment-13168811 ] Doug Cutting commented on HDFS-2664: I believe that HDFS-2647 is the best place for this discussion. That patch caused this test to fail. That issue should be reverted and re-committed when it can pass tests before it is committed. That's the normal practice. If you wish to change the contract for RPCEngine, if you wish to remove AvroRpcEngine, or however you wish to address it, it should be addressed before you commit rather than by committing a patch that causes existing tests to fail. If AvroRPCEngine is no longer of interest, remove it in a separate issue, otherwise maintain it so that its tests continue to pass. Please do not just remove the failing tests alone. Remove TestDFSOverAvroRpc - Key: HDFS-2664 URL: https://issues.apache.org/jira/browse/HDFS-2664 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 With HDFS-2647, HDFS has transitioned to protocol buffers. The server side implementation registers ProtocolPB.class and a BlockingService as implementation. Client side uses ProtocolPB.class as the interface. The RPC engine used is protobuf both for the RPC proxy and the server. With this TestDFSOverAvroRpc fails. I propose removing this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168823#comment-13168823 ] Suresh Srinivas commented on HDFS-2660: --- test common = test in common Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[
https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168821#comment-13168821
]
Scott Carey commented on HDFS-2660:
---
{quote}That was the intent. If that's no longer desired, then RPCEngine should
be removed and protobuf should be used directly. HDFS-2647 and now HDFS-2669
seem to be assuming that all RPC in HDFS will use protobuf and only protobuf,
since they're directly specifying protobuf-specifics in e.g.,
NameNodeRpcServer.java. That class used to be generic, indepdendent of any
particular RPC serialization or transport.
{quote}
It sounds like some modularity would help here. Perhaps use maven to put the
protobuf implementation into its own jar (and the avro implementation in its
own, etc) so that the core jar only contains the generic code and does not have
protobuf on its classpath. A user or distribution can select which
implementation to package and use.
Turn off TestDFSOverAvroRpc
---
Key: HDFS-2660
URL: https://issues.apache.org/jira/browse/HDFS-2660
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 0.24.0
Attachments: HDFS-2660.patch, HDFS-2660.txt
With HDFS-2647, protobuf based RPCs are enabled for some of the protocol.
With this, Avro RPC based protocol is not working. Avro based RPC needs to be
turned on similar to how protobuf based RPCs are done. Until such a time, I
propose turning off the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168822#comment-13168822 ] Suresh Srinivas commented on HDFS-2660: --- Doug, here is my proposal. I would like to remove Avro tests with HDFS and write a separate test common that tests AvroRPC, as done today with HDFS. Please vote if you agree, I will do this work. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2669) Enable protobuf rpc for ClientNamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168826#comment-13168826 ] Sanjay Radia commented on HDFS-2669: I would like to continue the discussion of TestDFSOverAvroRpc to HDFS-2660. Enable protobuf rpc for ClientNamenodeProtocol -- Key: HDFS-2669 URL: https://issues.apache.org/jira/browse/HDFS-2669 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: pbEnableClientNN1.patch, pbEnableClientNN4.patch Change the client and servers to use protobuf based RPC ClientNamenodeProtocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-2663: --- The last patch had some unnecessary changes. Will attach the right patch. Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168859#comment-13168859 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Common-trunk-Commit #1418 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1418/]) HDFS-2663. Optional protobuf parameters are not handled correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213981 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDfsOverAvroRpc.java Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-234: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks to Ivan Kelly for the great work! Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2663: -- Attachment: HDFS-2663.txt Attaching the right patch. Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168866#comment-13168866 ] Doug Cutting commented on HDFS-2660: There's already a test in common for Avro RPC that tests AvroRpcEngine. TestDFSOverAvroRpc tests that HDFS works over AvroRPCEngine. It did until HDFS-2647 was committed. Moreover the point of the Avro RPC support in Common is to support Avro RPC in HDFS. If that effort is to be abandoned then Avro RPC should be removed from Common too. Why else should we support it? If you want to just remove all of the Avro RPC stuff from Hadoop it should not be hard. There's AvroRpcEngine, AvroSpecificRpcEngine, TestAvroRpcEngine and AvroSpecificTestProtocol in common, TestDfsOverAvroRpc in HDFS, some annotations scattered around to help Avro reflection, and some residual stuff in YARN. Mostly you could probably just remove the avro-ipc dependency from various poms and then remove all the lines that the compiler complains about. That would get 90% of it. If you don't intend to keep TestDFSOverAvroRpc working as you develop HDFS then I think it would be best to remove Avro RPC support from Hadoop altogether. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168868#comment-13168868 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Hdfs-trunk-Commit #1493 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1493/]) HDFS-2663. Optional protobuf parameters are not handled correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213981 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDfsOverAvroRpc.java Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168869#comment-13168869 ] Hudson commented on HDFS-234: - Integrated in Hadoop-Hdfs-trunk-Commit #1493 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1493/]) HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213983 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties * /hadoop/common/trunk/hadoop-hdfs-project/pom.xml Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168870#comment-13168870 ] Hudson commented on HDFS-234: - Integrated in Hadoop-Common-trunk-Commit #1419 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1419/]) HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213983 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties * /hadoop/common/trunk/hadoop-hdfs-project/pom.xml Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar,
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168878#comment-13168878 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1443 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1443/]) HDFS-2663. Optional protobuf parameters are not handled correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213981 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDfsOverAvroRpc.java Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168879#comment-13168879 ] Hudson commented on HDFS-234: - Integrated in Hadoop-Mapreduce-trunk-Commit #1443 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1443/]) HDFS-234. Integration with BookKeeper logging system. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213983 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/README.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/EditLogLedgerMetadata.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/MaxTxId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/WriteLock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties * /hadoop/common/trunk/hadoop-hdfs-project/pom.xml Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar,
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168883#comment-13168883 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Hdfs-trunk-Commit #1494 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1494/]) HDFS-2663. Optional protobuf parameters are not handled correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213985 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168885#comment-13168885 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Common-trunk-Commit #1420 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1420/]) HDFS-2663. Optional protobuf parameters are not handled correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213985 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168895#comment-13168895 ] Hudson commented on HDFS-2663: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1444 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1444/]) HDFS-2663. Optional protobuf parameters are not handled correctly. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1213985 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto Optional parameters are not handled correctly - Key: HDFS-2663 URL: https://issues.apache.org/jira/browse/HDFS-2663 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2663.txt, HDFS-2663.txt, HDFS-2663.txt Some of the parameters are not defined as optional. Also optional parameters are not handled correctly. The converters should look for if the optional parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168897#comment-13168897 ] Eli Collins commented on HDFS-2660: --- bq. Moreover the point of the Avro RPC support in Common is to support Avro RPC in HDFS. If that effort is to be abandoned then Avro RPC should be removed from Common too. Why else should we support it? Makes sense to me. No one I'm aware of is pushing on Avro RPC integration or has plans to. In which case how about we create a feature branch for HADOOP-6659, remove the relevant changes from trunk/23 and put them in this branch so if someone wants to pick it up and close it out they can, this way we don't have to spend cycles in other development maintaining code that's not being actively developed. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2676) Remove Avro RPC
Remove Avro RPC --- Key: HDFS-2676 URL: https://issues.apache.org/jira/browse/HDFS-2676 Project: Hadoop HDFS Issue Type: Bug Reporter: Suresh Srinivas Assignee: Suresh Srinivas Please see the discussion in HDFS-2660 for more details. I have created a branch HADOOP-6659 to save the Avro work, if in the future some one wants to use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2660) Turn off TestDFSOverAvroRpc
[ https://issues.apache.org/jira/browse/HDFS-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168933#comment-13168933 ] Suresh Srinivas commented on HDFS-2660: --- Eli, per your comment, I have created a branch HADOOP-6659. I have also created two jiras HDFS-2676 and HADOOP-7920. Turn off TestDFSOverAvroRpc --- Key: HDFS-2660 URL: https://issues.apache.org/jira/browse/HDFS-2660 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2660.patch, HDFS-2660.txt With HDFS-2647, protobuf based RPCs are enabled for some of the protocol. With this, Avro RPC based protocol is not working. Avro based RPC needs to be turned on similar to how protobuf based RPCs are done. Until such a time, I propose turning off the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2677) HA: Web UI should indicate the NN state
HA: Web UI should indicate the NN state --- Key: HDFS-2677 URL: https://issues.apache.org/jira/browse/HDFS-2677 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Eli Collins The DFS web UI should indicate whether it's an active or standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread
[
https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168960#comment-13168960
]
Tsz Wo (Nicholas), SZE commented on HDFS-2589:
--
Hi Daryn,
It seems that it is better to change HftpFileSystem but not
DelegationTokenRenewer.
In WebHdfsFileSystem, the lazy initialization is implemented in
addRenewAction(..), which is the only method using DT_RENEWER. It is more lazy
-- we even don't have to create a DelegationTokenRenewer object if
addRenewAction(..) is not invoked.
{code}
//WebHdfsFileSystem
private static DelegationTokenRenewerWebHdfsFileSystem DT_RENEWER = null;
private static synchronized void addRenewAction(final WebHdfsFileSystem
webhdfs) {
if (DT_RENEWER == null) {
DT_RENEWER = new
DelegationTokenRenewerWebHdfsFileSystem(WebHdfsFileSystem.class);
DT_RENEWER.start();
}
DT_RENEWER.addRenewAction(webhdfs);
}
{code}
unnecessary hftp token fetch and renewal thread
---
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 1.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-2589.patch
Instantiation of the hftp filesystem is causing a token to be implicitly
created and added to a custom token renewal thread. With the new token
renewal feature in the JT, this causes the mapreduce
{{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled
token, and an explicit token) and leave a spurious renewal thread running.
This thread should not be running in the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and
to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
