[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163358#comment-13163358 ] Hadoop QA commented on HDFS-2629: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506216/HDFS-2629.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1645//console This message is automatically generated. > Implement protobuf service for InterDatanodeProtocol > > > Key: HDFS-2629 > URL: https://issues.apache.org/jira/browse/HDFS-2629 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Reporter: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2629.txt, HDFS-2629.txt > > > This jira adds implementation for InterDatanodeProtocol protobuf service > along the same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2629: -- Description: This jira adds implementation for InterDatanodeProtocol protobuf service along the same lines as HDFS-2181 > Implement protobuf service for InterDatanodeProtocol > > > Key: HDFS-2629 > URL: https://issues.apache.org/jira/browse/HDFS-2629 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Reporter: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2629.txt, HDFS-2629.txt > > > This jira adds implementation for InterDatanodeProtocol protobuf service > along the same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2629: -- Release Note: (was: This jira adds implementation for InterDatanodeProtocol protobuf service along the same lines as HDFS-2181 ) Status: Patch Available (was: Open) > Implement protobuf service for InterDatanodeProtocol > > > Key: HDFS-2629 > URL: https://issues.apache.org/jira/browse/HDFS-2629 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Reporter: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2629.txt, HDFS-2629.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2629: -- Attachment: HDFS-2629.txt New patch with tests attached. > Implement protobuf service for InterDatanodeProtocol > > > Key: HDFS-2629 > URL: https://issues.apache.org/jira/browse/HDFS-2629 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Reporter: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2629.txt, HDFS-2629.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163288#comment-13163288 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1395 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1395/]) HDFS-1580. Committing the test case, it was missed in earlier commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163287#comment-13163287 ] Hudson commented on HDFS-2618: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1395 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1395/]) HDFS-2618. Fix for the build failure caused by previous commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210746 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163284#comment-13163284 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Common-trunk-Commit #1371 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1371/]) HDFS-1580. Committing the test case, it was missed in earlier commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163281#comment-13163281 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk-Commit #1443 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1443/]) HDFS-1580. Committing the test case, it was missed in earlier commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210749 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163276#comment-13163276 ] Hudson commented on HDFS-2618: -- Integrated in Hadoop-Common-trunk-Commit #1370 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1370/]) HDFS-2618. Fix for the build failure caused by previous commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210746 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163274#comment-13163274 ] Hudson commented on HDFS-2618: -- Integrated in Hadoop-Hdfs-trunk-Commit #1442 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1442/]) HDFS-2618. Fix for the build failure caused by previous commit. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210746 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2545) Webhdfs: Support multiple namenodes in federation
[ https://issues.apache.org/jira/browse/HDFS-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163243#comment-13163243 ] Hadoop QA commented on HDFS-2545: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506199/h2545_20111205.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1643//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1643//console This message is automatically generated. > Webhdfs: Support multiple namenodes in federation > - > > Key: HDFS-2545 > URL: https://issues.apache.org/jira/browse/HDFS-2545 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2545_20111202.patch, h2545_20111205.patch > > > DatanodeWebHdfsMethods only talks to the default namenode. It won't work if > there are multiple namenodes in federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163237#comment-13163237 ] Hudson commented on HDFS-2618: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1393 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1393/]) HDFS-2618. Implement protobuf service for NamenodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210719 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/NamenodeProtocolProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockKey.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/NamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163222#comment-13163222 ] Hudson commented on HDFS-2618: -- Integrated in Hadoop-Hdfs-trunk-Commit #1440 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1440/]) HDFS-2618. Implement protobuf service for NamenodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210719 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/NamenodeProtocolProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockKey.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/NamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-2178: - Attachment: HDFS-2178.patch Corrected patch updated (my MacOS FileSystem is case-insensitive and the rename to uppercase FS failed from a Fs typo-ed failed). Also, remove unnecessary Maven repos from root POM. Trimmed trailing spaces in files in patch. Removed commented code in TestWebhdfsFileSsytem class. Following up the 'pure proxy' comment, Hoop also can proxy viewfs (with the mapping table in hoop instead of the client), thing that (IMU) webhdfs cannot no. Technically we could move Hoop under common. But Hoop depends on HDFS for testing which it may be seen as a circular dependency (i'm saying may because in practice it is not, but module dir organization would suggest that). > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163221#comment-13163221 ] Alejandro Abdelnur commented on HDFS-2178: -- test-patch seems to fail when a patch goes across projects (common/hdfs/mapreduce) or touches assemblies/project-dist. I'll open a JIRA on that. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163217#comment-13163217 ] Hudson commented on HDFS-2618: -- Integrated in Hadoop-Common-trunk-Commit #1368 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1368/]) HDFS-2618. Implement protobuf service for NamenodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210719 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/NamenodeProtocolProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockKey.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/NamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163213#comment-13163213 ] Jitendra Nath Pandey commented on HDFS-2618: +1. > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1787) "Not enough xcievers" error should propagate to client
[ https://issues.apache.org/jira/browse/HDFS-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh reassigned HDFS-1787: Assignee: (was: Jonathan Hsieh) Unassigning since I haven't had time to work on this and since infrastructure changed underneath it. > "Not enough xcievers" error should propagate to client > -- > > Key: HDFS-1787 > URL: https://issues.apache.org/jira/browse/HDFS-1787 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon > Labels: newbie > Fix For: 0.24.0 > > Attachments: hdfs-1787.2.patch, hdfs-1787.3.patch, hdfs-1787.3.patch, > hdfs-1787.5.patch, hdfs-1787.patch > > > We find that users often run into the default transceiver limits in the DN. > Putting aside the inherent issues with xceiver threads, it would be nice if > the "xceiver limit exceeded" error propagated to the client. Currently, > clients simply see an EOFException which is hard to interpret, and have to go > slogging through DN logs to find the underlying issue. > The data transfer protocol should be extended to either have a special error > code for "not enough xceivers" or should have some error code for generic > errors with which a string can be attached and propagated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2618: -- Attachment: HDFS-2618.txt attached patch addresses the comments. bq. rename NamenodeProtocolTranslatorPB to NamenodeProtocolClientSideTranslator? This is the convention that has been used so far. The javadoc does cover the information that it is client side translator. > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2545) Webhdfs: Support multiple namenodes in federation
[ https://issues.apache.org/jira/browse/HDFS-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2545: - Attachment: h2545_20111205.patch h2545_20111205.patch: added a new test with a 4-namenode cluster. > Webhdfs: Support multiple namenodes in federation > - > > Key: HDFS-2545 > URL: https://issues.apache.org/jira/browse/HDFS-2545 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2545_20111202.patch, h2545_20111205.patch > > > DatanodeWebHdfsMethods only talks to the default namenode. It won't work if > there are multiple namenodes in federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2618: -- Status: Open (was: Patch Available) > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163198#comment-13163198 ] Jitendra Nath Pandey commented on HDFS-2618: # Typo in NamenodeProtocolServerSideTranslatorPB.java : request.getRegistartion. # Since we have NamenodeProtocolServerSideTranslatorPB, does it make sense to rename NamenodeProtocolTranslatorPB to NamenodeProtocolClientSideTranslator? If we change that, we should revisit other client side translators as well. > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163190#comment-13163190 ] Hadoop QA commented on HDFS-2430: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.TestLease org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.hdfs.TestClientReportBadBlock org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.tools.TestGetGroups org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestAppendDifferentChecksum org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.TestDatanodeConfig org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean org.apache.hadoop.hdf
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163180#comment-13163180 ] Suresh Srinivas commented on HDFS-2618: --- javac warnings are due to using deprecated methods such as rollEditLog(). > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163169#comment-13163169 ] Aaron T. Myers commented on HDFS-2430: -- Something went majorly wrong with that test run - OOMs and "Unable to create native thread." I've also run a selection of the failed tests on my local box, and they all passed. I've just kicked Jenkins to re-run test-patch. We'll see how that goes. > The number of failed or low-resource volumes the NN can tolerate should be > configurable > --- > > Key: HDFS-2430 > URL: https://issues.apache.org/jira/browse/HDFS-2430 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2430.patch, HDFS-2430.patch, HDFS-2430.patch, > HDFS-2430.patch, HDFS-2430.patch > > > Currently the number of failed or low-resource volumes the NN can tolerate is > effectively hard-coded at 1. It would be nice if this were configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163164#comment-13163164 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Hdfs-trunk-Commit #1439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1439/]) HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163163#comment-13163163 ] Hudson commented on HDFS-2581: -- Integrated in Hadoop-Hdfs-trunk-Commit #1439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1439/]) HDFS-2581. Implement protobuf service for JournalProtocol. Contributed Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210657 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/overview.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolProtocolBuffers/overview.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163159#comment-13163159 ] Hadoop QA commented on HDFS-2618: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506177/HDFS-2618.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 35 javac compiler warnings (more than the trunk's current 33 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1640//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1640//console This message is automatically generated. > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163147#comment-13163147 ] Hadoop QA commented on HDFS-2430: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.TestLease org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.tools.TestDelegationTokenFetcher org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.fs.loadGenerator.TestLoadGenerator org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.fs.TestGlobPaths org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.fs.TestResolveHdfsSymlink org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.fs.TestFcHdfsPermission org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.TestListPathServlet org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.TestLocalDFS org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.tools.TestGetGroups
[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163144#comment-13163144 ] Eli Collins commented on HDFS-2430: --- +1 rebase on 1580 looks correct. > The number of failed or low-resource volumes the NN can tolerate should be > configurable > --- > > Key: HDFS-2430 > URL: https://issues.apache.org/jira/browse/HDFS-2430 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2430.patch, HDFS-2430.patch, HDFS-2430.patch, > HDFS-2430.patch, HDFS-2430.patch > > > Currently the number of failed or low-resource volumes the NN can tolerate is > effectively hard-coded at 1. It would be nice if this were configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2634) Standby needs to ingest latest edit logs before transitioning to active`
[
https://issues.apache.org/jira/browse/HDFS-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163145#comment-13163145
]
Todd Lipcon commented on HDFS-2634:
---
I also would like to add a safety check in {{initJournalsForWrite}} that makes
sure that under no circumstances do we get overlapping txid ranges. ie prevent
the following from happening:
{code}
$ ls -l /tmp/name-shared/current/
total 4104
-rw-r--r-- 1 todd todd 1048576 2011-12-05 14:41
edits_001-001
-rw-r--r-- 1 todd todd 1048576 2011-12-05 14:45
edits_002-002
-rw-r--r-- 1 todd todd 1048576 2011-12-05 14:58
edits_003-008
-rw-r--r-- 1 todd todd 1048576 2011-12-05 14:59
edits_inprogress_003
{code}
> Standby needs to ingest latest edit logs before transitioning to active`
>
>
> Key: HDFS-2634
> URL: https://issues.apache.org/jira/browse/HDFS-2634
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
>
> When the standby transitions to active state, it needs to _read_ the latest
> edit logs before it reopens them for write access. Currently, the transition
> calls {{stopStandbyServices}}, which stops the tailer, but doesn't read ahead
> to the very end. This ends up leaving the shared edits dir in an inconsistent
> state where we have overlapping transaction IDs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2634) Standby needs to ingest latest edit logs before transitioning to active`
Standby needs to ingest latest edit logs before transitioning to active`
Key: HDFS-2634
URL: https://issues.apache.org/jira/browse/HDFS-2634
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
When the standby transitions to active state, it needs to _read_ the latest
edit logs before it reopens them for write access. Currently, the transition
calls {{stopStandbyServices}}, which stops the tailer, but doesn't read ahead
to the very end. This ends up leaving the shared edits dir in an inconsistent
state where we have overlapping transaction IDs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2627) HA: determine DN's view of which NN is active based on heartbeat responses
[ https://issues.apache.org/jira/browse/HDFS-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163139#comment-13163139 ] Eli Collins commented on HDFS-2627: --- +1 latest patch lgtm. Filed HDFS-2633 for the poor naming of isAlive. > HA: determine DN's view of which NN is active based on heartbeat responses > -- > > Key: HDFS-2627 > URL: https://issues.apache.org/jira/browse/HDFS-2627 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2627-v1.txt, hdfs-2627-v2.txt > > > This is the first part of the design described in [this > comment|https://issues.apache.org/jira/browse/HDFS-1972?focusedCommentId=13160601&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13160601] > in HDFS-1972. When the DNs start, they should not consider either of the NNs > in a block pool to be the active one. Rather, the NNs should include their HA > state as part of the heartbeat response to the DN, and the DN will believe > whichever NN claims to be active at a higher transaction ID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2633) BPOfferService#isAlive is poorly named
BPOfferService#isAlive is poorly named -- Key: HDFS-2633 URL: https://issues.apache.org/jira/browse/HDFS-2633 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.0 Reporter: Eli Collins Priority: Minor Per HDFS-2627 the current implementation returns true even if one of the actor threads is dead. "The only non-test use case for isAlive seems to be from BlockPoolSliceScanner and DataBlockScanner, where they're really trying to figure out whether they should stop scanning the block pool. If the BPOS is connected to any NN at all (regardless of active/standby) it needs to report true so that the scanners don't stop running. It would be nice to clean up these calls and specify in their function name that they're only meant for use in tests" and annotate @VisibleForTesting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163129#comment-13163129 ] Eli Collins commented on HDFS-2178: --- Some feedback on latest patch: * The class and file name don't match for HttpFSParams * Why are the additions need in the root pom? Don't they're used (thankfully!) * Still some commented out code > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, > HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2625) HA: TestDfsOverAvroRpc failing after introduction of HeartbeatResponse type
[ https://issues.apache.org/jira/browse/HDFS-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163123#comment-13163123 ] Doug Cutting commented on HDFS-2625: FYI, an alternative would have been to add @Nullable to the method in the interface. > HA: TestDfsOverAvroRpc failing after introduction of HeartbeatResponse type > --- > > Key: HDFS-2625 > URL: https://issues.apache.org/jira/browse/HDFS-2625 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: HA branch (HDFS-1623) > > Attachments: hdfs-2625.txt > > > TestDfsOverAvroRpc is failing after HDFS-2616. The issue seems to be that we > sometimes fill in the "commands" list in HeartbeatResponse with null to > indicate no commands. This makes avro barf with a cryptic message about > unions of nulls of unions of arrays of nulls or something. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2430: - Attachment: HDFS-2430.patch Thanks a lot for the review, Eli. I've added an @throws comment per your suggestion. The attached patch is also rebased on trunk, which had a few conflicts since HDFS-1580 was committed. > The number of failed or low-resource volumes the NN can tolerate should be > configurable > --- > > Key: HDFS-2430 > URL: https://issues.apache.org/jira/browse/HDFS-2430 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2430.patch, HDFS-2430.patch, HDFS-2430.patch, > HDFS-2430.patch, HDFS-2430.patch > > > Currently the number of failed or low-resource volumes the NN can tolerate is > effectively hard-coded at 1. It would be nice if this were configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163120#comment-13163120 ] Hudson commented on HDFS-2581: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1392 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1392/]) HDFS-2581. Implement protobuf service for JournalProtocol. Contributed Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210657 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/overview.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolProtocolBuffers/overview.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163108#comment-13163108 ] Eli Collins commented on HDFS-2178: --- Btw any idea why test-patch keeps failing? The patch applies cleanly for me on trunk. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, > HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163106#comment-13163106 ] Eli Collins commented on HDFS-2178: --- Thanks. Will checkout the updated patch. bq. Regarding the 'pure proxy' question, as I've told Sanjay offline doing so it would limit Hoop to be a proxy for webhdfs. As it is today is a proxy to any Hadoop FileSystem implementation. If the proxy isn't HDFS specific then shouldn't this live in common? It's reasonable to proxy non-HDFS file systems but is there a need? IMO having protocol compatibility between webhdfs and httpfs seems more of a need (and easier to maintain via sharing implementation) than providing a proxy for eg s3n. bq. In addition, if you want a pure http proxy, you should use and off the shelf HTTP proxy, no need for a gateway. The reason we don't want a "pure" http proxy over webhdfs is that the proxy has hadoop-specific access control right? > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, > HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2618: -- Status: Patch Available (was: Open) > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2618: -- Attachment: HDFS-2618.txt > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt, HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2618) Implement protobuf service for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2618: -- Attachment: HDFS-2618.txt Attached patch with tests. > Implement protobuf service for NamenodeProtocol > --- > > Key: HDFS-2618 > URL: https://issues.apache.org/jira/browse/HDFS-2618 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, > HDFS-2618.txt > > > This jira adds implementation for NamenodeProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163088#comment-13163088 ] Hudson commented on HDFS-2581: -- Integrated in Hadoop-Common-trunk-Commit #1367 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1367/]) HDFS-2581. Implement protobuf service for JournalProtocol. Contributed Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210657 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/overview.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolProtocolBuffers/overview.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2581: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2627) HA: determine DN's view of which NN is active based on heartbeat responses
[ https://issues.apache.org/jira/browse/HDFS-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2627: -- Attachment: hdfs-2627-v2.txt Here's a cleaned up patch with better javadoc, etc. > HA: determine DN's view of which NN is active based on heartbeat responses > -- > > Key: HDFS-2627 > URL: https://issues.apache.org/jira/browse/HDFS-2627 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2627-v1.txt, hdfs-2627-v2.txt > > > This is the first part of the design described in [this > comment|https://issues.apache.org/jira/browse/HDFS-1972?focusedCommentId=13160601&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13160601] > in HDFS-1972. When the DNs start, they should not consider either of the NNs > in a block pool to be the active one. Rather, the NNs should include their HA > state as part of the heartbeat response to the DN, and the DN will believe > whichever NN claims to be active at a higher transaction ID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163078#comment-13163078 ] Sanjay Radia commented on HDFS-2581: +1 > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2627) HA: determine DN's view of which NN is active based on heartbeat responses
[
https://issues.apache.org/jira/browse/HDFS-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163049#comment-13163049
]
Todd Lipcon commented on HDFS-2627:
---
The only non-test use case for {{isAlive}} seems to be from
{{BlockPoolSliceScanner}} and {{DataBlockScanner}}, where they're really trying
to figure out whether they should stop scanning the block pool. If the BPOS is
connected to any NN at all (regardless of active/standby) it needs to report
{{true}} so that the scanners don't stop running.
It would be nice to clean up these calls and specify in their function name
that they're only meant for use in tests, but it seems out-of-scope for this
JIRA.
> HA: determine DN's view of which NN is active based on heartbeat responses
> --
>
> Key: HDFS-2627
> URL: https://issues.apache.org/jira/browse/HDFS-2627
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: hdfs-2627-v1.txt
>
>
> This is the first part of the design described in [this
> comment|https://issues.apache.org/jira/browse/HDFS-1972?focusedCommentId=13160601&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13160601]
> in HDFS-1972. When the DNs start, they should not consider either of the NNs
> in a block pool to be the active one. Rather, the NNs should include their HA
> state as part of the heartbeat response to the DN, and the DN will believe
> whichever NN claims to be active at a higher transaction ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163033#comment-13163033 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1391 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1391/]) HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163022#comment-13163022 ] Hadoop QA commented on HDFS-2178: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506154/HDFS-2178.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 103 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1639//console This message is automatically generated. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, > HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163011#comment-13163011 ] Hudson commented on HDFS-1580: -- Integrated in Hadoop-Common-trunk-Commit #1366 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1366/]) HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210602 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-2178: - Attachment: HDFS-2178.patch Integrating the rest of Eli's comments: * All Params classes are in a single outer class * All FSOperations are in a single outer class * Got rid of custom StringUtils, using functionality from the existing one Regarding documentation, the ServerSetup has the minimal props to get Hoop up and running and then references to its default.xml for the full list of properties (in this way the doc will not go out of sync) > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, > HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1580: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks to Ivan. > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162987#comment-13162987 ] Hadoop QA commented on HDFS-2581: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506140/HDFS-2581.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1638//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1638//console This message is automatically generated. > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2632) existing in_use.lock file is removed after failing to lock this file
existing in_use.lock file is removed after failing to lock this file Key: HDFS-2632 URL: https://issues.apache.org/jira/browse/HDFS-2632 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Environment: Scientific Linux 5.3 Reporter: Dan Bradley If an attempt is made to start the namenode when it is already running, an exception is generated on failure to lock in_use.lock. However, there is a bug: in_use.lock is deleted! After that, if another attempt is made to start the namenode, there is no in_use.lock file, so the new instance goes ahead and starts messing with the namenode state files. It eventually fails to bind to the TCP port, but it has already done damage by that time. Specifically, the 'edits' file being written to by the running instance is moved to 'previous.checkpoint' so all further transactions are lost when the HDFS service is next restarted. We observed a case of data loss because of this. This issue relates to HDFS-1690, but the problem in HDFS-1690 was stated in a way that is specific to -format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162961#comment-13162961 ] Alejandro Abdelnur commented on HDFS-2178: -- Thanks for reviewing the patch Eli, Regarding the 'pure proxy' question, as I've told Sanjay offline doing so it would limit Hoop to be a proxy for webhdfs. As it is today is a proxy to any Hadoop FileSystem implementation. In addition, if you want a pure http proxy, you should use and off the shelf HTTP proxy, no need for a gateway. Regarding webhdfs/httpfs interop, the TestWebhdfsFileSystem test does exactly this. Regarding config keys, I was keeping things as isolated as possible from the rest of HDFS until Webhdfs & Hoop share the codebase. Regarding a code sharing JIRA, as soon as this JIRA goes in. All the other comments, I'll integrate them in a follow up patch. Thanks again. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2581: -- Attachment: HDFS-2581.txt Patch generated from the root. > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2581: -- Attachment: HDFS-2581.txt Tests included in the new patch. > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162942#comment-13162942 ] Eli Collins commented on HDFS-2178: --- @Tucu, Looking good. Sanjay's suggestion of having it be a pure proxy (forward operations to webhdfs) makes a lot of sense to me, plan to do that? I didn't look at param* and fsoperations* since I checked out a previous version and they mostly look identical in the latest patch. It would be easier to get a sense of all the concrete params and fsoperations classes lived in a single file, perhaps as nested classes? Where is webhdfs/httpfs interoperability tested? Seems like a good test would be to send streams of operations through a tee to both and assert the final contents of the file system are identical. Config keys should live in DFSConfigKeys next to the WebHDFS ones, eg the various "authentication.kerberos.keytab" defines. Could webhdfs and httpfs share config keys? User-visible stuff like config we should share asap so we don't break people when introducing the sharing. Feel free to handle in a follow-on jira. File a jira for code sharing with WebHDFS? All the config keys should be covered in ServerSetup.apt.vm right? Is the info there sufficient to setup a proxy w security enabled? Nit: remove commented out code, eg TestWebHDFSFileSytem#operations Nit: Why not re-use/update core's StringUtils instead of introducing a new one? Thanks, Eli > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, > HDFS-2178.patch, HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2215) Add toString to FileStatus
[ https://issues.apache.org/jira/browse/HDFS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162930#comment-13162930 ] Hadoop QA commented on HDFS-2215: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506131/HDFS-2215.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1637//console This message is automatically generated. > Add toString to FileStatus > -- > > Key: HDFS-2215 > URL: https://issues.apache.org/jira/browse/HDFS-2215 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: SreeHari >Priority: Minor > Attachments: HDFS-2215.patch > > > It would be nice if FileStatus had a reasonable toString, for debugging > purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2215) Add toString to FileStatus
[ https://issues.apache.org/jira/browse/HDFS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HDFS-2215: - Assignee: SreeHari > Add toString to FileStatus > -- > > Key: HDFS-2215 > URL: https://issues.apache.org/jira/browse/HDFS-2215 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: SreeHari >Priority: Minor > Attachments: HDFS-2215.patch > > > It would be nice if FileStatus had a reasonable toString, for debugging > purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2606) webhdfs client filesystem impl must set the content-type header for create/append
[
https://issues.apache.org/jira/browse/HDFS-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162921#comment-13162921
]
Alejandro Abdelnur commented on HDFS-2606:
--
The thing is that the Jersey is processing eagerly that content-type as form
input.
The correct thing to do is that the server should reject a request that does
not have a application/octet-stream content type. This ensure the data is not
modified in transit by proxies. Other content-types (such as text based ones)
don't guarantee this.
> webhdfs client filesystem impl must set the content-type header for
> create/append
> -
>
> Key: HDFS-2606
> URL: https://issues.apache.org/jira/browse/HDFS-2606
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.24.0, 0.23.1
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>Priority: Critical
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2606.patch
>
>
> Currently the content-type header is not being set and for some reason for
> append it is being set to the form encoded content type making jersey
> parameter parsing fail.
> For this and to avoid any kind of proxy transcoding the content-type should
> be set to binary.
> {code}
> conn.setRequestProperty("Content-Type", "application/octet-stream");
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2215) Add toString to FileStatus
[ https://issues.apache.org/jira/browse/HDFS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-2215: -- Status: Patch Available (was: Open) > Add toString to FileStatus > -- > > Key: HDFS-2215 > URL: https://issues.apache.org/jira/browse/HDFS-2215 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: SreeHari >Priority: Minor > Attachments: HDFS-2215.patch > > > It would be nice if FileStatus had a reasonable toString, for debugging > purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2215) Add toString to FileStatus
[
https://issues.apache.org/jira/browse/HDFS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162911#comment-13162911
]
HariSree commented on HDFS-2215:
Attached the patch .
toString implementation will be (sample)-
For File :
FileStatus{path=/testfile; isDirectory=false; length=1234; replication=3;
blocksize=67108864; owner=root; group=supergroup; permission=--xr--r--}
For Directory :
FileStatus{path=/testfile; isDirectory=true; owner=root; group=supergroup;
permission=--xr--r--}
Did not attach test case since doesn't seem to add any value for this change .
If required , can add one .
> Add toString to FileStatus
> --
>
> Key: HDFS-2215
> URL: https://issues.apache.org/jira/browse/HDFS-2215
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jakob Homan
>Priority: Minor
> Attachments: HDFS-2215.patch
>
>
> It would be nice if FileStatus had a reasonable toString, for debugging
> purposes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2215) Add toString to FileStatus
[ https://issues.apache.org/jira/browse/HDFS-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HariSree updated HDFS-2215: --- Attachment: HDFS-2215.patch > Add toString to FileStatus > -- > > Key: HDFS-2215 > URL: https://issues.apache.org/jira/browse/HDFS-2215 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jakob Homan >Priority: Minor > Attachments: HDFS-2215.patch > > > It would be nice if FileStatus had a reasonable toString, for debugging > purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2627) HA: determine DN's view of which NN is active based on heartbeat responses
[ https://issues.apache.org/jira/browse/HDFS-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162905#comment-13162905 ] Eli Collins commented on HDFS-2627: --- Sorry I meant AND, ie BPOfferService#isAlive is not alive if its thread talking to the primary is dead right? > HA: determine DN's view of which NN is active based on heartbeat responses > -- > > Key: HDFS-2627 > URL: https://issues.apache.org/jira/browse/HDFS-2627 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2627-v1.txt > > > This is the first part of the design described in [this > comment|https://issues.apache.org/jira/browse/HDFS-1972?focusedCommentId=13160601&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13160601] > in HDFS-1972. When the DNs start, they should not consider either of the NNs > in a block pool to be the active one. Rather, the NNs should include their HA > state as part of the heartbeat response to the DN, and the DN will believe > whichever NN claims to be active at a higher transaction ID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2631) Rewrite fuse-dfs to use the webhdfs protocol
Rewrite fuse-dfs to use the webhdfs protocol Key: HDFS-2631 URL: https://issues.apache.org/jira/browse/HDFS-2631 Project: Hadoop HDFS Issue Type: Improvement Components: contrib/fuse-dfs Reporter: Eli Collins We should port the implementation of fuse-dfs to use the webhdfs protocol. This has a number of benefits: * Compatibility - allows a single fuse client to work across server versions * Works with both WebHDFS and Hoop since they are protocol compatible * Removes the overhead related to libhdfs (forking a jvm) * Makes it easier to support features like security -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162881#comment-13162881 ] Hadoop QA commented on HDFS-2581: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506124/HDFS-2581.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1636//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1636//console This message is automatically generated. > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2581: -- Attachment: HDFS-2581.txt > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2630) Chance for Stream leakage in CompressorStream
Chance for Stream leakage in CompressorStream - Key: HDFS-2630 URL: https://issues.apache.org/jira/browse/HDFS-2630 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0 Reporter: HariSree Priority: Minor Fix For: 0.23.0 In CompressorStream.close , finish() can throw IOException . But out will not be closed in that situation since it is not in finally -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol
[ https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162681#comment-13162681 ] Hadoop QA commented on HDFS-2581: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506095/HDFS-2581.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1635//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1635//console This message is automatically generated. > Implement protobuf service for JournalProtocol > -- > > Key: HDFS-2581 > URL: https://issues.apache.org/jira/browse/HDFS-2581 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, > HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt > > > This jira adds implementation for JournalProtocol protobuf service along the > same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
