date:20111206

[
https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163545#comment-13163545
]

Hadoop QA commented on HDFS-234:

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506247/HDFS-234.diff
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The patch appears to cause tar ant target to fail.

-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed the unit tests build

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1646//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1646//console

This message is automatically generated.

Integration with BookKeeper logging system
--

Key: HDFS-234
URL: https://issues.apache.org/jira/browse/HDFS-234
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Luca Telloli
Assignee: Ivan Kelly
Fix For: HA branch (HDFS-1623), 0.24.0

Attachments: HADOOP-5189-trunk-preview.patch,
HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch,
HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff,
HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar,
zookeeper-dev.jar

BookKeeper is a system to reliably log streams of records
(https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a
natural target for such a system for being the metadata repository of the
entire file system for HDFS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163563#comment-13163563
 ] 

Hudson commented on HDFS-2581:
--

Integrated in Hadoop-Hdfs-trunk #886 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/886/])
HDFS-2581. Implement protobuf service for JournalProtocol. Contributed 
Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210657
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/overview.html
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolProtocolBuffers/overview.html
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for JournalProtocol
 --

 Key: HDFS-2581
 URL: https://issues.apache.org/jira/browse/HDFS-2581
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, 
 HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, 
 HDFS-2581.txt


 This jira adds implementation for JournalProtocol protobuf service along the 
 same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163564#comment-13163564
 ] 

Hudson commented on HDFS-2618:
--

Integrated in Hadoop-Hdfs-trunk #886 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/886/])
HDFS-2618. Fix for the build failure caused by previous commit.
HDFS-2618. Implement protobuf service for NamenodeProtocol. Contributed by 
Suresh Srinivas.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210746
Files : 
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210719
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/NamenodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockKey.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/NamenodeProtocol.proto
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for NamenodeProtocol
 ---

 Key: HDFS-2618
 URL: https://issues.apache.org/jira/browse/HDFS-2618
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, 
 HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt


 This jira adds implementation for NamenodeProtocol protobuf service along the 
 same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms


[ 
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163565#comment-13163565
 ] 

Hudson commented on HDFS-1580:
--

Integrated in Hadoop-Hdfs-trunk #886 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/886/])
HDFS-1580. Committing the test case, it was missed in earlier commit.
HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. 
Contributed by Ivan Kelly.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749
Files : 
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java


 Add interface for generic Write Ahead Logging mechanisms
 

 Key: HDFS-1580
 URL: https://issues.apache.org/jira/browse/HDFS-1580
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Jitendra Nath Pandey
 Fix For: HA branch (HDFS-1623), 0.24.0

 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, 
 EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, 
 HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, 
 HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, 
 HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, 
 generic_wal_iface.pdf, generic_wal_iface.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms


[ 
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163600#comment-13163600
 ] 

Hudson commented on HDFS-1580:
--

Integrated in Hadoop-Mapreduce-trunk #919 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/919/])
HDFS-1580. Committing the test case, it was missed in earlier commit.
HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. 
Contributed by Ivan Kelly.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210749
Files : 
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210602
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java


 Add interface for generic Write Ahead Logging mechanisms
 

 Key: HDFS-1580
 URL: https://issues.apache.org/jira/browse/HDFS-1580
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Jitendra Nath Pandey
 Fix For: HA branch (HDFS-1623), 0.24.0

 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, 
 EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, 
 HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, 
 HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, 
 HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, 
 generic_wal_iface.pdf, generic_wal_iface.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2581) Implement protobuf service for JournalProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163598#comment-13163598
 ] 

Hudson commented on HDFS-2581:
--

Integrated in Hadoop-Mapreduce-trunk #919 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/919/])
HDFS-2581. Implement protobuf service for JournalProtocol. Contributed 
Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210657
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/overview.html
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolProtocolBuffers/overview.html
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for JournalProtocol
 --

 Key: HDFS-2581
 URL: https://issues.apache.org/jira/browse/HDFS-2581
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, 
 HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, HDFS-2581.txt, 
 HDFS-2581.txt


 This jira adds implementation for JournalProtocol protobuf service along the 
 same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2618) Implement protobuf service for NamenodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163599#comment-13163599
 ] 

Hudson commented on HDFS-2618:
--

Integrated in Hadoop-Mapreduce-trunk #919 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/919/])
HDFS-2618. Fix for the build failure caused by previous commit.
HDFS-2618. Implement protobuf service for NamenodeProtocol. Contributed by 
Suresh Srinivas.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210746
Files : 
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1210719
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/NamenodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockKey.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/NamenodeProtocol.proto
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for NamenodeProtocol
 ---

 Key: HDFS-2618
 URL: https://issues.apache.org/jira/browse/HDFS-2618
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, 
 HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt


 This jira adds implementation for NamenodeProtocol protobuf service along the 
 same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities

[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163694#comment-13163694
]

Eli Collins commented on HDFS-2178:
---

bq. Technically we could move Hoop under common. But Hoop depends on HDFS for
testing which it may be seen as a circular dependency (i'm saying may because
in practice it is not, but module dir organization would suggest that).

If it's not hdfs-specific then common is the right place. We have the same
issue with other non-hdfs-specific features that live in common (eg the client
side mount table and symlinks). In terms of test structure you can make
TestHttpFSFileSystem a base class and have TestHDFSBasedHttpFileSystem and
TestWebhdfsBasedFileSystem live in HDFS. Make sense?

Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
capabilities
--

Key: HDFS-2178
URL: https://issues.apache.org/jira/browse/HDFS-2178
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 0.24.0, 0.23.1

Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFSoverHTTP-API.html, HdfsHttpAPI.pdf

We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement)
for HDFS Proxy.
Hoop provides access to all Hadoop Distributed File System (HDFS) operations
(read and write) over HTTP/S.
The Hoop server component is a REST HTTP gateway to HDFS supporting all file
system operations. It can be accessed using standard HTTP tools (i.e. curl
and wget), HTTP libraries from different programing languages (i.e. Perl,
Java Script) as well as using the Hoop client. The Hoop server component is a
standard Java web-application and it has been implemented using Jersey
(JAX-RS).
The Hoop client component is an implementation of Hadoop FileSystem client
that allows using the familiar Hadoop filesystem API to access HDFS data
through a Hoop server.
Repo: https://github.com/cloudera/hoop
Docs: http://cloudera.github.com/hoop
Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/
Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for
Kerberos HTTP SPNEGO authentication).
To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be
done first, as well as the Alfredo contribution (HADOOP-7119).

[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


 [ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2629:
--

Attachment: HDFS-2629.txt

New patch generated from the base directory instead of hadoop-hdfs.

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol

[
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163702#comment-13163702
]

Hadoop QA commented on HDFS-2629:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506280/HDFS-2629.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The patch appears to cause tar ant target to fail.

-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed the unit tests build

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1647//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1647//console

This message is automatically generated.

Implement protobuf service for InterDatanodeProtocol

Key: HDFS-2629
URL: https://issues.apache.org/jira/browse/HDFS-2629
Project: Hadoop HDFS
Issue Type: New Feature
Components: data-node
Reporter: Suresh Srinivas
Fix For: 0.24.0

Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt

This jira adds implementation for InterDatanodeProtocol protobuf service
along the same lines as HDFS-2181

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities

2011-12-06 Thread Alejandro Abdelnur (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163712#comment-13163712
]

Alejandro Abdelnur commented on HDFS-2178:
--

I've missed something before, Hoop actually depends on the HDFS JAR. HDFS JAR
has to go into its WAR.

Regarding having testcases in a diff module than the classes being tested, it
may be confusing/misleading to developers that testcases are not present. Plus
changes won't be tested if you are working on that module (only when running
the full tests from trunk or the test of another module). Granted, much of the
tests we are doing should be integration tests (and in this case, yes they
should go into another module, but it is not how we have things today).

Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
capabilities
--

Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFSoverHTTP-API.html, HdfsHttpAPI.pdf

[jira] [Updated] (HDFS-1765) Block Replication should respect under-replication block priority

2011-12-06 Thread Uma Maheswara Rao G (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-1765:
--

Attachment: HDFS-1765.patch

 Block Replication should respect under-replication block priority
 -

 Key: HDFS-1765
 URL: https://issues.apache.org/jira/browse/HDFS-1765
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 0.23.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
 Fix For: 0.24.0

 Attachments: HDFS-1765.patch, HDFS-1765.pdf, underReplicatedQueue.pdf


 Currently under-replicated blocks are assigned different priorities depending 
 on how many replicas a block has. However the replication monitor works on 
 blocks in a round-robin fashion. So the newly added high priority blocks 
 won't get replicated until all low-priority blocks are done. One example is 
 that on decommissioning datanode WebUI we often observe that blocks with 
 only decommissioning replicas do not get scheduled to replicate before other 
 blocks, so risking data availability if the node is shutdown for repair 
 before decommission completes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1765) Block Replication should respect under-replication block priority

2011-12-06 Thread Uma Maheswara Rao G (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Uma Maheswara Rao G updated HDFS-1765:
--

Status: Patch Available (was: Open)

HDFS-1765.patch :1st Patch.
Updated with the propose solution.

@Nicholas, Addressed your comment for moving the chooseUnderReplicatedBlocks to
UnderReplicatedBlocks class.
Thanks a lot for the document Reviews.

Block Replication should respect under-replication block priority
-

Key: HDFS-1765
URL: https://issues.apache.org/jira/browse/HDFS-1765
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.23.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.24.0

Attachments: HDFS-1765.patch, HDFS-1765.pdf, underReplicatedQueue.pdf

Currently under-replicated blocks are assigned different priorities depending
on how many replicas a block has. However the replication monitor works on
blocks in a round-robin fashion. So the newly added high priority blocks
won't get replicated until all low-priority blocks are done. One example is
that on decommissioning datanode WebUI we often observe that blocks with
only decommissioning replicas do not get scheduled to replicate before other
blocks, so risking data availability if the node is shutdown for repair
before decommission completes.

[jira] [Commented] (HDFS-1765) Block Replication should respect under-replication block priority

[
https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163749#comment-13163749
]

Hadoop QA commented on HDFS-1765:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506283/HDFS-1765.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1648//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1648//console

This message is automatically generated.

Block Replication should respect under-replication block priority
-

Attachments: HDFS-1765.patch, HDFS-1765.pdf, underReplicatedQueue.pdf

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities

[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163753#comment-13163753
]

Eli Collins commented on HDFS-2178:
---

bq. Hoop actually depends on the HDFS JAR. HDFS JAR has to go into its WAR.

That would be true for other file system implementations as well right? IMO
either Hoop is cross hadoop file system and we test/support that use case and
should put it in common to prevent it from becoming hdfs specific, or it's hdfs
specific in which case it should live in HDFS and proxy to webhdfs.

Wrt test coverage if a change is made to Hoop in common that breaks eg HDFS or
WebHDFS backends then those jenkins jobs fail as soon as they pull in the new
artifact. This is how it works today for other cross-component features.

Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
capabilities
--

Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFSoverHTTP-API.html, HdfsHttpAPI.pdf

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities

2011-12-06 Thread Alejandro Abdelnur (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163779#comment-13163779
]

Alejandro Abdelnur commented on HDFS-2178:
--

Hoop primary use case is to proxy to HDFS. The DistributedFileSystem client
implementation different from other filesystem implementations, it is not in
hadoop-common but in hadoop-hdfs (the same goes for WebHdfsFileSystem client
implementation). Should these implementations be moved to hadoop-common?

Regarding Hoop being a proxy webhdfs, you could use a HTTP proxy for that. But
this means you'd have to configure you (whatever) client to use the HTTP proxy.
And all cluster nodes names would be exposed. Or you could make a specialized
proxy, this would be a reverse proxy. This would add another network hop and it
would not work with other filesystems implementations.

Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
capabilities
--

Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFSoverHTTP-API.html, HdfsHttpAPI.pdf

[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


 [ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2629:
--

Attachment: HDFS-2629.txt

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2635) NNBench creates duplicate files if multiple maps are run from the same client

2011-12-06 Thread cp...@kence.org (Created) (JIRA)

NNBench creates duplicate files if multiple maps are run from the same client
-

 Key: HDFS-2635
 URL: https://issues.apache.org/jira/browse/HDFS-2635
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.205.0, 0.20.204.0, 0.20.203.0
Reporter: cp...@kence.org
Priority: Minor


NNBench creates files in the format:
  file_hostname__filenum

This works seamlessly as long as all of the Hadoop clients in the cluster are 
each running with a single map slot. If multiple map slots are available on a 
single client, each mapper tries to create the same set of files. This can 
result in lock contention on some non-Hadoop HDFS implementations thereby 
defeating the purpose of the NNBench test.

Making the files unique per mapper, not per host, resolves this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2635) NNBench creates duplicate files if multiple maps are run from the same client

2011-12-06 Thread cp...@kence.org (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

cp...@kence.org updated HDFS-2635:
--

Target Version/s: 0.20.203.0
  Status: Patch Available  (was: Open)

--- NNBench.java2011-12-06 12:03:49.0 -0800
+++ NNBench.new 2011-12-06 12:03:31.0 -0800
@@ -722,16 +722,16 @@
   if (barrier()) {
 if (op.equals(OP_CREATE_WRITE)) {
   startTimeTPmS = System.currentTimeMillis();
-  doCreateWriteOp(file_ + hostName + _, reporter);
+  doCreateWriteOp(file_ + hostName + _ + key, reporter);
 } else if (op.equals(OP_OPEN_READ)) {
   startTimeTPmS = System.currentTimeMillis();
-  doOpenReadOp(file_ + hostName + _, reporter);
+  doOpenReadOp(file_ + hostName + _ + key, reporter);
 } else if (op.equals(OP_RENAME)) {
   startTimeTPmS = System.currentTimeMillis();
-  doRenameOp(file_ + hostName + _, reporter);
+  doRenameOp(file_ + hostName + _ + key, reporter);
 } else if (op.equals(OP_DELETE)) {
   startTimeTPmS = System.currentTimeMillis();
-  doDeleteOp(file_ + hostName + _, reporter);
+  doDeleteOp(file_ + hostName + _ + key, reporter);
 }
 
 endTimeTPms = System.currentTimeMillis();

 NNBench creates duplicate files if multiple maps are run from the same client
 -

 Key: HDFS-2635
 URL: https://issues.apache.org/jira/browse/HDFS-2635
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.205.0, 0.20.204.0, 0.20.203.0
Reporter: cp...@kence.org
Priority: Minor

 NNBench creates files in the format:
   file_hostname__filenum
 This works seamlessly as long as all of the Hadoop clients in the cluster are 
 each running with a single map slot. If multiple map slots are available on a 
 single client, each mapper tries to create the same set of files. This can 
 result in lock contention on some non-Hadoop HDFS implementations thereby 
 defeating the purpose of the NNBench test.
 Making the files unique per mapper, not per host, resolves this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2635) NNBench creates duplicate files if multiple maps are run from the same client

2011-12-06 Thread cp...@kence.org (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

cp...@kence.org updated HDFS-2635:
--

Status: Open  (was: Patch Available)

 NNBench creates duplicate files if multiple maps are run from the same client
 -

 Key: HDFS-2635
 URL: https://issues.apache.org/jira/browse/HDFS-2635
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.205.0, 0.20.204.0, 0.20.203.0
Reporter: cp...@kence.org
Priority: Minor

 NNBench creates files in the format:
   file_hostname__filenum
 This works seamlessly as long as all of the Hadoop clients in the cluster are 
 each running with a single map slot. If multiple map slots are available on a 
 single client, each mapper tries to create the same set of files. This can 
 result in lock contention on some non-Hadoop HDFS implementations thereby 
 defeating the purpose of the NNBench test.
 Making the files unique per mapper, not per host, resolves this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2635) NNBench creates duplicate files if multiple maps are run from the same client

2011-12-06 Thread cp...@kence.org (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

cp...@kence.org updated HDFS-2635:
--

Attachment: hdfs-2635.patch

Lets try this again, the 'right' way.

 NNBench creates duplicate files if multiple maps are run from the same client
 -

 Key: HDFS-2635
 URL: https://issues.apache.org/jira/browse/HDFS-2635
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.203.0, 0.20.204.0, 0.20.205.0
Reporter: cp...@kence.org
Priority: Minor
 Attachments: hdfs-2635.patch


 NNBench creates files in the format:
   file_hostname__filenum
 This works seamlessly as long as all of the Hadoop clients in the cluster are 
 each running with a single map slot. If multiple map slots are available on a 
 single client, each mapper tries to create the same set of files. This can 
 result in lock contention on some non-Hadoop HDFS implementations thereby 
 defeating the purpose of the NNBench test.
 Making the files unique per mapper, not per host, resolves this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2635) NNBench creates duplicate files if multiple maps are run from the same client

2011-12-06 Thread cp...@kence.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163802#comment-13163802
 ] 

cp...@kence.org commented on HDFS-2635:
---

My submitted patch fixes the file duplication by incorporating the mapper's key 
into the filename. The combination of hostname and map key result in a unique 
filename across multiple hosts with a variable number of map slots per host.

 NNBench creates duplicate files if multiple maps are run from the same client
 -

 Key: HDFS-2635
 URL: https://issues.apache.org/jira/browse/HDFS-2635
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.20.203.0, 0.20.204.0, 0.20.205.0
Reporter: cp...@kence.org
Priority: Minor
 Attachments: hdfs-2635.patch


 NNBench creates files in the format:
   file_hostname__filenum
 This works seamlessly as long as all of the Hadoop clients in the cluster are 
 each running with a single map slot. If multiple map slots are available on a 
 single client, each mapper tries to create the same set of files. This can 
 result in lock contention on some non-Hadoop HDFS implementations thereby 
 defeating the purpose of the NNBench test.
 Making the files unique per mapper, not per host, resolves this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol

[
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163829#comment-13163829
]

Hadoop QA commented on HDFS-2629:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506301/HDFS-2629.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1649//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1649//console

This message is automatically generated.

Implement protobuf service for InterDatanodeProtocol

Key: HDFS-2629
URL: https://issues.apache.org/jira/browse/HDFS-2629
Project: Hadoop HDFS
Issue Type: New Feature
Components: data-node
Reporter: Suresh Srinivas
Fix For: 0.24.0

Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt,
HDFS-2629.txt

This jira adds implementation for InterDatanodeProtocol protobuf service
along the same lines as HDFS-2181

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities


[ 
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163832#comment-13163832
 ] 

Eli Collins commented on HDFS-2178:
---

Per offline conversation with Tucu:
# It makes sense to keep Hoop proxying directly to Hadoop file systems because 
this allows us to support federation and a proxy (ie proxy to viewfs vs viewfs 
mapping to a proxy) and because http redirecting to webhdfs won't work for 
clients that are firewalled off from datanodes.
# Ideally the code could live in common (or a top-level project in a flat 
hierarchy if we move to that) but this would require multiple mvn projects as 
we'd need one in common and one in hdfs to generate the war with the injected 
hdfs jar and the build pain isn't worth it. 

Think the current approach makes sense.

 Contributing Hoop to HDFS, replacement for HDFS proxy with read/write 
 capabilities
 --

 Key: HDFS-2178
 URL: https://issues.apache.org/jira/browse/HDFS-2178
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 0.24.0, 0.23.1

 Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, 
 HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, 
 HDFSoverHTTP-API.html, HdfsHttpAPI.pdf


 We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) 
 for HDFS Proxy.
 Hoop provides access to all Hadoop Distributed File System (HDFS) operations 
 (read and write) over HTTP/S.
 The Hoop server component is a REST HTTP gateway to HDFS supporting all file 
 system operations. It can be accessed using standard HTTP tools (i.e. curl 
 and wget), HTTP libraries from different programing languages (i.e. Perl, 
 Java Script) as well as using the Hoop client. The Hoop server component is a 
 standard Java web-application and it has been implemented using Jersey 
 (JAX-RS).
 The Hoop client component is an implementation of Hadoop FileSystem client 
 that allows using the familiar Hadoop filesystem API to access HDFS data 
 through a Hoop server.
   Repo: https://github.com/cloudera/hoop
   Docs: http://cloudera.github.com/hoop
   Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/
 Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for 
 Kerberos HTTP SPNEGO authentication). 
 To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be 
 done first, as well as the Alfredo contribution (HADOOP-7119).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol

2011-12-06 Thread Jitendra Nath Pandey (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163845#comment-13163845
]

Hadoop QA commented on HDFS-2629:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506301/HDFS-2629.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1650//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1650//console

This message is automatically generated.

Implement protobuf service for InterDatanodeProtocol

Key: HDFS-2629
URL: https://issues.apache.org/jira/browse/HDFS-2629
Project: Hadoop HDFS
Issue Type: New Feature
Components: data-node
Reporter: Suresh Srinivas
Fix For: 0.24.0

Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt,
HDFS-2629.txt

This jira adds implementation for InterDatanodeProtocol protobuf service
along the same lines as HDFS-2181

[jira] [Commented] (HDFS-2565) hdfs command does not explicitely complain that the command I am trying to run is not valid

2011-12-06 Thread Tom White (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163847#comment-13163847
 ] 

Tom White commented on HDFS-2565:
-

This is the same behaviour as other bin scripts, like 'hadoop' and 'mapred', so 
any fix should be for all of them.

 hdfs command does not explicitely complain that the command I am trying to 
 run is not valid
 ---

 Key: HDFS-2565
 URL: https://issues.apache.org/jira/browse/HDFS-2565
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
  Labels: bigtop, noob

 [root@bigtop-fedora-15 ~]# hdfs foobar
 Exception in thread main java.lang.NoClassDefFoundError: foobar
 Caused by: java.lang.ClassNotFoundException: foobar
 at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
 Could not find the main class: foobar. Program will exit.
 Instead of loading any class, it would be nice to explain the command is not 
 valid and to call print_usage()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities

2011-12-06 Thread Alejandro Abdelnur (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163881#comment-13163881
]

Alejandro Abdelnur commented on HDFS-2178:
--

Thanks Eli.

All tests are passing, including the one that validates interoperability with
the Webhdfs file system.

Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
capabilities
--

Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch,
HDFSoverHTTP-API.html, HdfsHttpAPI.pdf

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163904#comment-13163904
 ] 

Jitendra Nath Pandey commented on HDFS-2629:


# Javadoc in InterDatanodeProtocolServersideTranslator refers to JournalProtocl.

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol

2011-12-06 Thread Suresh Srinivas (Created) (JIRA)

Implement protobuf service for ClientDatanodeProtocol
-

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas


This jira adds implementation for InterDatanodeProtocol protobuf service along 
the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2618) Implement protobuf service for NamenodeProtocol

2011-12-06 Thread Suresh Srinivas (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HDFS-2618.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

I committed the patch.

 Implement protobuf service for NamenodeProtocol
 ---

 Key: HDFS-2618
 URL: https://issues.apache.org/jira/browse/HDFS-2618
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt, 
 HDFS-2618.txt, HDFS-2618.txt, HDFS-2618.txt


 This jira adds implementation for NamenodeProtocol protobuf service along the 
 same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol

2011-12-06 Thread Jitendra Nath Pandey (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2629:
--

Attachment: HDFS-2629.txt

New patch addresses the comments

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol

2011-12-06 Thread Suresh Srinivas (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HDFS-2629.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

I committed the patch.

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163914#comment-13163914
 ] 

Jitendra Nath Pandey commented on HDFS-2629:


+1

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


 [ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2629:
--

Status: Open  (was: Patch Available)

 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163918#comment-13163918
 ] 

Hudson commented on HDFS-2629:
--

Integrated in Hadoop-Hdfs-trunk-Commit #1445 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1445/])
HDFS-2629. Implement protobuf service for InterDatanodeProtocol. 
Contributed by Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1211206
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/InterDatanodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/InterDatanodeProtocol.proto
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163920#comment-13163920
 ] 

Hudson commented on HDFS-2629:
--

Integrated in Hadoop-Common-trunk-Commit #1373 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1373/])
HDFS-2629. Implement protobuf service for InterDatanodeProtocol. 
Contributed by Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1211206
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/InterDatanodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/InterDatanodeProtocol.proto
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2629) Implement protobuf service for InterDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163954#comment-13163954
 ] 

Hudson commented on HDFS-2629:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #1397 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1397/])
HDFS-2629. Implement protobuf service for InterDatanodeProtocol. 
Contributed by Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1211206
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/InterDatanodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/JournalProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/InterDatanodeProtocol.proto
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for InterDatanodeProtocol
 

 Key: HDFS-2629
 URL: https://issues.apache.org/jira/browse/HDFS-2629
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node
Reporter: Suresh Srinivas
 Fix For: 0.24.0

 Attachments: HDFS-2629.txt, HDFS-2629.txt, HDFS-2629.txt, 
 HDFS-2629.txt, HDFS-2629.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2637) The rpc timeout for block recovery is too low

2011-12-06 Thread Eli Collins (Created) (JIRA)

The rpc timeout for block recovery is too low 
--

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins


The RPC timeout for block recovery does not take into account that it issues 
multiple RPCs itself. This can cause recovery to fail if the network is 
congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2638) Improve a blog recovery log

2011-12-06 Thread Eli Collins (Created) (JIRA)

Improve a blog recovery log
---

 Key: HDFS-2638
 URL: https://issues.apache.org/jira/browse/HDFS-2638
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor


It would be useful to know whether an attempt to recover a block is failing 
because the block was already recovered (has a new GS) or the block is missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2639) A client may fail during block recovery even if its request to recover a block succeeds

2011-12-06 Thread Eli Collins (Created) (JIRA)

A client may fail during block recovery even if its request to recover a block 
succeeds
---

 Key: HDFS-2639
 URL: https://issues.apache.org/jira/browse/HDFS-2639
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins


The client gets stuck in the following loop if an rpc its issued to recover a 
block timed out:

{noformat}
DataStreamer#run
1.  processDatanodeError
2. DN#recoverBlock
3.DN#syncBlock
4.   NN#nextGenerationStamp
5.  sleep 1s
6.  goto 1
{noformat}

Once we've timed out onece at step 2 and loop, step 2 throws an IOE because the 
block is already being recovered and step 4 throws an IOE because the block GS 
is now out of date (the previous, timed-out, request got a new GS and updated 
the block). Eventually the client reaches max retries, considers all DNs bad, 
and close throws an IOE.

The client should be able to succeed if one of its requests to recover the 
block succeeded. It should still fail if another client (eg HBase via 
recoverLease or the NN via releaseLease) succesfully recovered the block. One 
way to handle this would be to not timeout the request to recover the block. 
Another would be able to make a subsequent call to recoverBlock succeed eg by 
updating the block's sequence number to be the latest value that was updated by 
the same client in the previous request (ie it can recover over itself but not 
another client).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol


 [ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2636:
--

Attachment: HDFS-2636.txt

 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol


 [ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2636:
--

Status: Patch Available  (was: Open)

 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2637) The rpc timeout for block recovery is too low

2011-12-06 Thread Eli Collins (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins updated HDFS-2637:
--

Attachment: hdfs-2637-b1-1.patch

Patch attached.

 The rpc timeout for block recovery is too low 
 --

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2637-b1-1.patch


 The RPC timeout for block recovery does not take into account that it issues 
 multiple RPCs itself. This can cause recovery to fail if the network is 
 congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2638) Improve a blog recovery log

2011-12-06 Thread Eli Collins (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins updated HDFS-2638:
--

Attachment: hdfs-2638-b1-1.patch

Patch attached.

 Improve a blog recovery log
 ---

 Key: HDFS-2638
 URL: https://issues.apache.org/jira/browse/HDFS-2638
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
 Attachments: hdfs-2638-b1-1.patch


 It would be useful to know whether an attempt to recover a block is failing 
 because the block was already recovered (has a new GS) or the block is 
 missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol

[
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163999#comment-13163999
]

Hadoop QA commented on HDFS-2636:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506331/HDFS-2636.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1651//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1651//console

This message is automatically generated.

Implement protobuf service for ClientDatanodeProtocol
-

Key: HDFS-2636
URL: https://issues.apache.org/jira/browse/HDFS-2636
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Attachments: HDFS-2636.txt

This jira adds implementation for InterDatanodeProtocol protobuf service
along the same lines as HDFS-2181

[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities

2011-12-06 Thread Jitendra Nath Pandey (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164023#comment-13164023
 ] 

Eli Collins commented on HDFS-2178:
---

Latest patch looks good, mostly small stuff:
* In TestHdfsHelper looks like you can nuke the shutdown hook? (see this is 
hanging in addShutdownHook)
* Conditionals, catches etc go on the same line as the preceding brace (eg  } 
catch {, } else {) eg in FSOperations, HttpFSFileSystem
* Some variables have underbars (eg new_dir)
* Don't need Empty class javadoc headers eg TestHdfsHelper
* Some spelling mistakes: thrwon, aboslute, whe the test
* Can remove the code in Uncomment to verify things fail in TestHTestCase and 
TestHFSTestCase or modify the tests to actually run and handle the failure 
(doesn't seem necessary)

 Contributing Hoop to HDFS, replacement for HDFS proxy with read/write 
 capabilities
 --

 Key: HDFS-2178
 URL: https://issues.apache.org/jira/browse/HDFS-2178
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 0.24.0, 0.23.1

 Attachments: HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, 
 HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, HDFS-2178.patch, 
 HDFSoverHTTP-API.html, HdfsHttpAPI.pdf


 We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) 
 for HDFS Proxy.
 Hoop provides access to all Hadoop Distributed File System (HDFS) operations 
 (read and write) over HTTP/S.
 The Hoop server component is a REST HTTP gateway to HDFS supporting all file 
 system operations. It can be accessed using standard HTTP tools (i.e. curl 
 and wget), HTTP libraries from different programing languages (i.e. Perl, 
 Java Script) as well as using the Hoop client. The Hoop server component is a 
 standard Java web-application and it has been implemented using Jersey 
 (JAX-RS).
 The Hoop client component is an implementation of Hadoop FileSystem client 
 that allows using the familiar Hadoop filesystem API to access HDFS data 
 through a Hoop server.
   Repo: https://github.com/cloudera/hoop
   Docs: http://cloudera.github.com/hoop
   Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/
 Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for 
 Kerberos HTTP SPNEGO authentication). 
 To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be 
 done first, as well as the Alfredo contribution (HADOOP-7119).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164024#comment-13164024
 ] 

Jitendra Nath Pandey commented on HDFS-2636:


+1, lgtm.

 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2637) The rpc timeout for block recovery is too low


[ 
https://issues.apache.org/jira/browse/HDFS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164025#comment-13164025
 ] 

Eli Collins commented on HDFS-2637:
---

Unit tests are running, earlier run was clean. test-patch results follow. 4 
findbugs warnings are HADOOP-7847, not this patch.

{noformat}
 [exec] 
 [exec] -1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] -1 tests included.  The patch doesn't appear to include any new 
or modified tests.
 [exec] Please justify why no tests are needed for 
this patch.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] -1 findbugs.  The patch appears to introduce 4 new Findbugs 
(version 1.3.9) warnings.
 [exec] 
{noformat}

 The rpc timeout for block recovery is too low 
 --

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2637-b1-1.patch


 The RPC timeout for block recovery does not take into account that it issues 
 multiple RPCs itself. This can cause recovery to fail if the network is 
 congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2638) Improve a blog recovery log


[ 
https://issues.apache.org/jira/browse/HDFS-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164026#comment-13164026
 ] 

Eli Collins commented on HDFS-2638:
---

Unit tests are running, earlier run was clean. test-patch results follow. 4 
findbugs warnings are HADOOP-7847, not this patch. Existing test coverage 
should be sufficient.

{noformat}
 [exec] 
 [exec] -1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] -1 tests included.  The patch doesn't appear to include any new 
or modified tests.
 [exec] Please justify why no tests are needed for 
this patch.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] -1 findbugs.  The patch appears to introduce 4 new Findbugs 
(version 1.3.9) warnings.
 [exec] 
{noformat}

 Improve a blog recovery log
 ---

 Key: HDFS-2638
 URL: https://issues.apache.org/jira/browse/HDFS-2638
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
 Attachments: hdfs-2638-b1-1.patch


 It would be useful to know whether an attempt to recover a block is failing 
 because the block was already recovered (has a new GS) or the block is 
 missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol


 [ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2636:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I committed the patch.

 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164029#comment-13164029
 ] 

Hudson commented on HDFS-2636:
--

Integrated in Hadoop-Hdfs-trunk-Commit #1447 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1447/])
HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. 
Contributed by Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1211249
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/ClientDatanodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/ClientDatanodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/InterDatanodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol


[ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164030#comment-13164030
 ] 

Hudson commented on HDFS-2636:
--

Integrated in Hadoop-Common-trunk-Commit #1375 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1375/])
HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. 
Contributed by Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1211249
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/ClientDatanodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/ClientDatanodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/InterDatanodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2636) Implement protobuf service for ClientDatanodeProtocol

2011-12-06 Thread Uma Maheswara Rao G (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164038#comment-13164038
 ] 

Hudson commented on HDFS-2636:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #1399 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1399/])
HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. 
Contributed by Suresh Srinivas.

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1211249
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/ClientDatanodeProtocolProtos.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/InterDatanodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/ClientDatanodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/InterDatanodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java


 Implement protobuf service for ClientDatanodeProtocol
 -

 Key: HDFS-2636
 URL: https://issues.apache.org/jira/browse/HDFS-2636
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2636.txt


 This jira adds implementation for InterDatanodeProtocol protobuf service 
 along the same lines as HDFS-2181

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable

2011-12-06 Thread Aaron T. Myers (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164048#comment-13164048
 ] 

Aaron T. Myers commented on HDFS-2430:
--

Same sort of odd failure. I've just kicked Jenkins again. Third time's a charm?

 The number of failed or low-resource volumes the NN can tolerate should be 
 configurable
 ---

 Key: HDFS-2430
 URL: https://issues.apache.org/jira/browse/HDFS-2430
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Attachments: HDFS-2430.patch, HDFS-2430.patch, HDFS-2430.patch, 
 HDFS-2430.patch, HDFS-2430.patch


 Currently the number of failed or low-resource volumes the NN can tolerate is 
 effectively hard-coded at 1. It would be nice if this were configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2637) The rpc timeout for block recovery is too low

2011-12-06 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164059#comment-13164059
 ] 

Todd Lipcon commented on HDFS-2637:
---

+1

 The rpc timeout for block recovery is too low 
 --

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2637-b1-1.patch


 The RPC timeout for block recovery does not take into account that it issues 
 multiple RPCs itself. This can cause recovery to fail if the network is 
 congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2638) Improve a block recovery log

2011-12-06 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-2638:
--

Summary: Improve a block recovery log  (was: Improve a blog recovery log)

 Improve a block recovery log
 

 Key: HDFS-2638
 URL: https://issues.apache.org/jira/browse/HDFS-2638
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
 Attachments: hdfs-2638-b1-1.patch


 It would be useful to know whether an attempt to recover a block is failing 
 because the block was already recovered (has a new GS) or the block is 
 missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2638) Improve a block recovery log

2011-12-06 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164062#comment-13164062
 ] 

Todd Lipcon commented on HDFS-2638:
---

+1

 Improve a block recovery log
 

 Key: HDFS-2638
 URL: https://issues.apache.org/jira/browse/HDFS-2638
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
 Attachments: hdfs-2638-b1-1.patch


 It would be useful to know whether an attempt to recover a block is failing 
 because the block was already recovered (has a new GS) or the block is 
 missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1765) Block Replication should respect under-replication block priority

2011-12-06 Thread Uma Maheswara Rao G (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G updated HDFS-1765:
--

Attachment: HDFS-1765.patch

 Block Replication should respect under-replication block priority
 -

 Key: HDFS-1765
 URL: https://issues.apache.org/jira/browse/HDFS-1765
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 0.23.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
 Fix For: 0.24.0

 Attachments: HDFS-1765.patch, HDFS-1765.patch, HDFS-1765.pdf, 
 underReplicatedQueue.pdf


 Currently under-replicated blocks are assigned different priorities depending 
 on how many replicas a block has. However the replication monitor works on 
 blocks in a round-robin fashion. So the newly added high priority blocks 
 won't get replicated until all low-priority blocks are done. One example is 
 that on decommissioning datanode WebUI we often observe that blocks with 
 only decommissioning replicas do not get scheduled to replicate before other 
 blocks, so risking data availability if the node is shutdown for repair 
 before decommission completes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2637) The rpc timeout for block recovery is too low


[ 
https://issues.apache.org/jira/browse/HDFS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164091#comment-13164091
 ] 

Uma Maheswara Rao G commented on HDFS-2637:
---

This may be applicable for trunk as well right?
I think this issue is same as HDFS-2378. Can it be duplicated?

 The rpc timeout for block recovery is too low 
 --

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2637-b1-1.patch


 The RPC timeout for block recovery does not take into account that it issues 
 multiple RPCs itself. This can cause recovery to fail if the network is 
 congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2637) The rpc timeout for block recovery is too low

2011-12-06 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164093#comment-13164093
 ] 

Todd Lipcon commented on HDFS-2637:
---

doh, yea, this is dup of HDFS-2378. Somehow when Eli and I were discussing it 
this morning I forgot that I'd already filed a JIRA for it. Since we already 
have the patch here, we can just commit this one and mark the other as dup, I 
guess?

In trunk the block recovery algorithm is quite different so I don't think it 
applies -- the block recovery is triggered as part of the pipeline setup, 
rather than an RPC. But maybe we have a similar type of bug in trunk with 
timeouts in the pipeline setup process? Mind taking a look, Uma?

 The rpc timeout for block recovery is too low 
 --

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2637-b1-1.patch


 The RPC timeout for block recovery does not take into account that it issues 
 multiple RPCs itself. This can cause recovery to fail if the network is 
 congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable

2011-12-06 Thread Uma Maheswara Rao G (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164099#comment-13164099
]

Hadoop QA commented on HDFS-2430:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.TestParallelRead
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
org.apache.hadoop.hdfs.TestQuota
org.apache.hadoop.hdfs.security.token.block.TestBlockToken
org.apache.hadoop.hdfs.TestFileAppend3
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.tools.TestGetGroups

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1652//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1652//console

This message is automatically generated.

The number of failed or low-resource volumes the NN can tolerate should be
configurable
---

Key: HDFS-2430
URL: https://issues.apache.org/jira/browse/HDFS-2430
Project: Hadoop HDFS
Issue Type: New Feature
Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HDFS-2430.patch, HDFS-2430.patch, HDFS-2430.patch,
HDFS-2430.patch, HDFS-2430.patch

Currently the number of failed or low-resource volumes the NN can tolerate is
effectively hard-coded at 1. It would be nice if this were configurable.

[jira] [Commented] (HDFS-2637) The rpc timeout for block recovery is too low


[ 
https://issues.apache.org/jira/browse/HDFS-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164100#comment-13164100
 ] 

Uma Maheswara Rao G commented on HDFS-2637:
---

Ok Todd, i will take a look on trunk code once and update if it really applies.
Why i pointed for trunk is because HDFS-2378 filed against to 0.23.0 and 1.1.0.

 The rpc timeout for block recovery is too low 
 --

 Key: HDFS-2637
 URL: https://issues.apache.org/jira/browse/HDFS-2637
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 1.0.0
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hdfs-2637-b1-1.patch


 The RPC timeout for block recovery does not take into account that it issues 
 multiple RPCs itself. This can cause recovery to fail if the network is 
 congested or DNs are busy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2378) recoverBlock timeout in DFSClient should be longer

2011-12-06 Thread Uma Maheswara Rao G (Resolved) (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Uma Maheswara Rao G resolved HDFS-2378.
---

Resolution: Duplicate

Since the patch in HDFS-2637 already got +1.
After discussing with Todd, this can be duplicated. Marking it as duplicate of
HDFS-2637.

recoverBlock timeout in DFSClient should be longer
--

Key: HDFS-2378
URL: https://issues.apache.org/jira/browse/HDFS-2378
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.23.0, 1.1.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
Fix For: 0.24.0

In a failure scenario when one of the datanodes in a pipeline has frozen
(eg hard swapping or disk controller issues) we sometimes see timeouts in the
call to recoverBlock(). This is because recoverBlock's implementation sends
several RPCs internally (to the NN and to other nodes in the pipeline) with
the same timeout. Since the timeouts are equal, the outer call times out
first. The retry then fails since recovery is already in progress, or already
finished.
The best fix would be to make recoverBlock idempotent so the retry doesn't
fail, but in the absence of that we can likely fix this issue by increasing
the timeout to be equal to the sum of the timeouts of the underlying recovery
calls.

[jira] [Commented] (HDFS-1765) Block Replication should respect under-replication block priority