[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.

2011-09-16 Thread Uma Maheswara Rao G (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105893#comment-13105893
 ] 

Uma Maheswara Rao G commented on HDFS-2271:
---

Hi Arun,

Thanks for taking a look!

 This Code  design has been refactored in trunk. ( as part of HDFS-1073).
 Most of the cases handled with HDFS-1073.
 I am checking the trunk code considering this scenarios, if i find such cases 
i will file the JIRAs. 

Thanks
Uma

 startJournalSpool should invoke ProcessIOError with failed storage 
 directories if createEditLogFile throws any exception.  
 ---

 Key: HDFS-2271
 URL: https://issues.apache.org/jira/browse/HDFS-2271
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 0.22.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
 Fix For: 0.22.0

 Attachments: HDFS-2271.1.patch, HDFS-2271.1.patch, HDFS-2271.patch


 Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, 
 BackUPNode will proceed with exceptions. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Aaron T. Myers (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105896#comment-13105896
 ] 

Aaron T. Myers commented on HDFS-2337:
--

No tests are included since this is basically just a refactor. The fact that 
underlying resources are closed when a {{RetryInvocationHandler}} which wraps 
another {{InvocationHandler}} is closed was tested in HADOOP-7635.

The failing tests are unrelated.

 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Sanjay Radia (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105917#comment-13105917
 ] 

Sanjay Radia commented on HDFS-2318:


Comments on h2318_20110915.patch. Looks good - here are my comments. 
* Namenode:  Change   //add jersey resource to // add webhdfs via jersey
*  jspHelper#getUgi  - don't like //try ugi parameter code. Instead overload 
method
* jspHelper#getUgi  setAuthenticationMethod  to KERBEROS and not KERBEROS_SSL
* If user param is specified it must match delegation token or the kerberos 
user name. (I think the old code had a bug since it does not check this). Later 
we can add the case for proxy.
* AuthFilter - for PseudoAuthentication you set anonymous to true - why.  (you 
have correctly used the default false for Kerberos.) Previously you had to set 
it true for tests to pass. Given that the auth filter is now for a specific 
path, does this still need to be set to true for tests to pass?


 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, h2318_20110915b_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Sanjay Radia (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13105944#comment-13105944
 ] 

Sanjay Radia commented on HDFS-2318:


Thanks for addressing my comments. Comments on h2318_20110915b.patch 
 If user param is specified it must match delegation token or the kerberos 
 user name.
This needs to be done for both delegation token and also for kerberos; you did 
it only for delegation token. Hence  user needs to checked against the Kerberos 
principal.

+1 except for the above.


 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, h2318_20110915b_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2338) Configuration option to enable/disable webhdfs.

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106032#comment-13106032
 ] 

Hudson commented on HDFS-2338:
--

Integrated in Hadoop-Hdfs-trunk #802 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/802/])
HDFS-2338. Add configuration option to enable/disable webhdfs.  Contributed 
by jitendra

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171379
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java


 Configuration option to enable/disable webhdfs.
 ---

 Key: HDFS-2338
 URL: https://issues.apache.org/jira/browse/HDFS-2338
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
 Fix For: 0.20.205.0, 0.24.0

 Attachments: HDFS-2338.1.patch, HDFS-2338.20s.1.patch, 
 HDFS-2338.20s.2.patch


 We should add a configuration option to enable/disable webhdfs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2333) HDFS-2284 introduced 2 findbugs warnings on trunk

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106034#comment-13106034
 ] 

Hudson commented on HDFS-2333:
--

Integrated in Hadoop-Hdfs-trunk #802 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/802/])
HDFS-2333. Change DFSOutputStream back to package private, otherwise, there 
are two SC_START_IN_CTOR findbugs warnings.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171136
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java


 HDFS-2284 introduced 2 findbugs warnings on trunk
 -

 Key: HDFS-2333
 URL: https://issues.apache.org/jira/browse/HDFS-2333
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Tsz Wo (Nicholas), SZE
 Fix For: 0.20.205.0, 0.24.0

 Attachments: HDFS-2333.diff, h2333_20110914.patch, 
 h2333_20110914b.patch, h2333_20110915.patch, h2333_20110915_0.20s.patch


 When HDFS-2284 was submitted it made DFSOutputStream public which triggered 
 two SC_START_IN_CTOR findbug warnings.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Tsz Wo (Nicholas), SZE (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-2318:
-

Attachment: h2318_20110916.patch
h2318_20110916_0.20s.patch

h2318_20110916_0.20s.patch
h2318_20110916.patch

Added username check for the other cases.

 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, 
 h2318_20110915b_0.20s.patch, h2318_20110916.patch, h2318_20110916_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2338) Configuration option to enable/disable webhdfs.

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106050#comment-13106050
 ] 

Hudson commented on HDFS-2338:
--

Integrated in Hadoop-Mapreduce-trunk #832 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/832/])
HDFS-2338. Add configuration option to enable/disable webhdfs.  Contributed 
by jitendra

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171379
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java


 Configuration option to enable/disable webhdfs.
 ---

 Key: HDFS-2338
 URL: https://issues.apache.org/jira/browse/HDFS-2338
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
 Fix For: 0.20.205.0, 0.24.0

 Attachments: HDFS-2338.1.patch, HDFS-2338.20s.1.patch, 
 HDFS-2338.20s.2.patch


 We should add a configuration option to enable/disable webhdfs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2333) HDFS-2284 introduced 2 findbugs warnings on trunk

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106052#comment-13106052
 ] 

Hudson commented on HDFS-2333:
--

Integrated in Hadoop-Mapreduce-trunk #832 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/832/])
HDFS-2333. Change DFSOutputStream back to package private, otherwise, there 
are two SC_START_IN_CTOR findbugs warnings.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171136
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java


 HDFS-2284 introduced 2 findbugs warnings on trunk
 -

 Key: HDFS-2333
 URL: https://issues.apache.org/jira/browse/HDFS-2333
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Tsz Wo (Nicholas), SZE
 Fix For: 0.20.205.0, 0.24.0

 Attachments: HDFS-2333.diff, h2333_20110914.patch, 
 h2333_20110914b.patch, h2333_20110915.patch, h2333_20110915_0.20s.patch


 When HDFS-2284 was submitted it made DFSOutputStream public which triggered 
 two SC_START_IN_CTOR findbug warnings.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2339) BackUpNode is not getting shutdown/recover when all volumes failed

2011-09-16 Thread Uma Maheswara Rao G (JIRA) 
BackUpNode is not getting shutdown/recover when all volumes failed
--

 Key: HDFS-2339
 URL: https://issues.apache.org/jira/browse/HDFS-2339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 0.24.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


When all volumes failed at back up node side, it is waiting for ever.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2339) BackUpNode is not getting shutdown/recover when all volumes failed

2011-09-16 Thread Uma Maheswara Rao G (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106096#comment-13106096
 ] 

Uma Maheswara Rao G commented on HDFS-2339:
---

It looks to me that backup node is waiting when all volumes failed.

   {code}
 LOG.info(Waiting until the NameNode rolls its edit logs in order  +
to freeze the BackupNode namespace.);
while (bnState == BNState.IN_SYNC) {
  Preconditions.checkState(stopApplyingEditsOnNextRoll,
If still in sync, we should still have the flag set to  +
freeze at next roll);
  try {
wait();
  } catch (InterruptedException ie) {
LOG.warn(Interrupted waiting for namespace to freeze, ie);
throw new IOException(ie);
  }
}
   {code}
   
  NameNode already removed the back up journals on failure. So, it is not 
giving any call to BNN.

  Since the backupnode streams already insync before , it is just waiting. 
  Why do we need to run the process at this stage?


 BackUpNode is not getting shutdown/recover when all volumes failed
 --

 Key: HDFS-2339
 URL: https://issues.apache.org/jira/browse/HDFS-2339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 0.24.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G

 When all volumes failed at back up node side, it is waiting for ever.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2339) BackUpNode is not getting shutdown/recover when all volumes failed

2011-09-16 Thread Uma Maheswara Rao G (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106098#comment-13106098
 ] 

Uma Maheswara Rao G commented on HDFS-2339:
---

Some more info:

11/09/16 19:31:24 INFO namenode.FSEditLog: Number of transactions: 6 Total time 
for transactions(ms): 2Number of transactions batched in Syncs: 19 Number of 
syncs: 6 SyncTimes(ms): 77
11/09/16 19:32:45 INFO namenode.FSEditLog: Number of transactions: 8 Total time 
for transactions(ms): 2Number of transactions batched in Syncs: 19 Number of 
syncs: 8 SyncTimes(ms): 77
11/09/16 19:34:42 INFO namenode.FSEditLog: Number of transactions: 10 Total 
time for transactions(ms): 60401Number of transactions batched in Syncs: 19 
Number of syncs: 10 SyncTimes(ms): 77
11/09/16 19:35:40 INFO namenode.FSImage: NameNode started a new log segment at 
txid 137
11/09/16 19:35:40 INFO namenode.FSEditLog: Ending log segment 121
11/09/16 19:35:40 INFO namenode.FSEditLog: Number of transactions: 11 Total 
time for transactions(ms): 61080Number of transactions batched in Syncs: 19 
Number of syncs: 12 SyncTimes(ms): 77
11/09/16 19:35:40 ERROR namenode.FSEditLog: Error ending log segment (journal 
JournalAndStream(mgr=FileJournalManager(root=/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08),
 
stream=/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08/current/edits_inprogress_121))
java.io.IOException: Unable to finalize edits file 
/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08/current/edits_inprogress_121
at 
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:97)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog$JournalAndStream.close(FSEditLog.java:1209)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog$JournalAndStream.access$4(FSEditLog.java:1202)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog$4.apply(FSEditLog.java:880)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.mapJournalsAndReportErrors(FSEditLog.java:1049)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:876)
at 
org.apache.hadoop.hdfs.server.namenode.BackupImage.namenodeStartedLogSegment(BackupImage.java:355)
at 
org.apache.hadoop.hdfs.server.namenode.BackupNode$BackupNodeRpcServer.startLogSegment(BackupNode.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:632)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
11/09/16 19:35:40 ERROR namenode.FSEditLog: Disabling journal 
...
...
JournalAndStream(mgr=FileJournalManager(root=/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08),
 stream=null)
11/09/16 19:35:41 INFO ipc.Server: IPC Server handler 0 on 50100, call: 
startLogSegment(NamenodeRegistration(HOST-10-18-52-222:9000, role=NameNode), 
137), rpc version=2, client version=1, methodsFingerPrint=-852377201 from 
10.18.52.222:43158, error:
java.io.IOException: Unable to start log segment 137: no journals successfully 
started.
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegment(FSEditLog.java:843)
at 
org.apache.hadoop.hdfs.server.namenode.BackupImage.namenodeStartedLogSegment(BackupImage.java:370)
at 
org.apache.hadoop.hdfs.server.namenode.BackupNode$BackupNodeRpcServer.startLogSegment(BackupNode.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:632)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at

[jira] [Created] (HDFS-2340) Support getContentSummary, getFileChecksum and getDelegationToken in webhdfs

2011-09-16 Thread Tsz Wo (Nicholas), SZE (JIRA) 
Support getContentSummary, getFileChecksum and getDelegationToken in webhdfs


 Key: HDFS-2340
 URL: https://issues.apache.org/jira/browse/HDFS-2340
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106500#comment-13106500
 ] 

Hadoop QA commented on HDFS-2318:
-

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12494791/h2318_20110916.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 10 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed these unit tests:
  org.apache.hadoop.hdfs.TestDfsOverAvroRpc
  
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/1259//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1259//console

This message is automatically generated.

 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, 
 h2318_20110915b_0.20s.patch, h2318_20110916.patch, h2318_20110916_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Tsz Wo (Nicholas), SZE (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-2318:
-

   Resolution: Fixed
Fix Version/s: 0.24.0
   0.20.205.0
 Hadoop Flags: [Reviewed]
   Status: Resolved  (was: Patch Available)

The failed tests are not related.

I have committed this.

 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Fix For: 0.20.205.0, 0.24.0

 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, 
 h2318_20110915b_0.20s.patch, h2318_20110916.patch, h2318_20110916_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2243) DataXceiver per accept seems to be a bottleneck in HBase/YCSB test

2011-09-16 Thread Eric Caspole (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Caspole updated HDFS-2243:
---

Fix Version/s: 0.24.0
   Status: Patch Available  (was: Open)

 DataXceiver per accept seems to be a bottleneck in HBase/YCSB test
 --

 Key: HDFS-2243
 URL: https://issues.apache.org/jira/browse/HDFS-2243
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
 Environment: Using Fedora 14 on a quad core phenom system
Reporter: Eric Caspole
Priority: Minor
 Fix For: 0.24.0

 Attachments: HDFS-2234-branch-0.20-append.patch, 
 HDFS-2243-0.23-110909.txt, datanode-perf-110808.gif


 I am running the YCSB benchmark against HBase, sometimes against a single 
 node, sometimes against a cluster of 6 systems. As the load increases into 
 thousands of TPS, especially on the single node, I can see that the datanode 
 runs very high system time and seems to be bottlenecked by how fast it can 
 create the threads to handle the new connections in DataXceiverServer.run. By 
 perf top I can see the process spends about 12% of all its time in 
 pthread_create, and in hprof profiles I can see there are tens of thousands 
 of threads created in just a few minutes of test execution.
 Does anyone else observe this bottleneck? Is there a major challenge to using 
 a thread pool of DataXceivers in this situation?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106510#comment-13106510
 ] 

Hudson commented on HDFS-2318:
--

Integrated in Hadoop-Hdfs-trunk-Commit #976 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/976/])
Revised the entry for HDFS-2318 in CHANGES.txt.
HDFS-2318. Provide authentication to webhdfs using SPNEGO and delegation tokens.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171615
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171611
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetDelegationTokenServlet.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/KerberosUgiAuthenticator.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BlockSizeParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DelegationParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/HttpOpParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserProvider.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java


 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Fix For: 0.20.205.0, 0.24.0

 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, 
 h2318_20110915b_0.20s.patch, h2318_20110916.patch, h2318_20110916_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106509#comment-13106509
 ] 

Hudson commented on HDFS-2318:
--

Integrated in Hadoop-Common-trunk-Commit #899 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/899/])
Revised the entry for HDFS-2318 in CHANGES.txt.
HDFS-2318. Provide authentication to webhdfs using SPNEGO and delegation tokens.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171615
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171611
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetDelegationTokenServlet.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/KerberosUgiAuthenticator.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BlockSizeParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DelegationParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/HttpOpParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserProvider.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java


 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Fix For: 0.20.205.0, 0.24.0

 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, 
 h2318_20110915b_0.20s.patch, h2318_20110916.patch, h2318_20110916_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2243) DataXceiver per accept seems to be a bottleneck in HBase/YCSB test

2011-09-16 Thread Eric Caspole (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Caspole updated HDFS-2243:
---

Status: Open  (was: Patch Available)

 DataXceiver per accept seems to be a bottleneck in HBase/YCSB test
 --

 Key: HDFS-2243
 URL: https://issues.apache.org/jira/browse/HDFS-2243
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
 Environment: Using Fedora 14 on a quad core phenom system
Reporter: Eric Caspole
Priority: Minor
 Fix For: 0.24.0

 Attachments: HDFS-2234-branch-0.20-append.patch, 
 HDFS-2243-0.23-110909.txt, datanode-perf-110808.gif


 I am running the YCSB benchmark against HBase, sometimes against a single 
 node, sometimes against a cluster of 6 systems. As the load increases into 
 thousands of TPS, especially on the single node, I can see that the datanode 
 runs very high system time and seems to be bottlenecked by how fast it can 
 create the threads to handle the new connections in DataXceiverServer.run. By 
 perf top I can see the process spends about 12% of all its time in 
 pthread_create, and in hprof profiles I can see there are tens of thousands 
 of threads created in just a few minutes of test execution.
 Does anyone else observe this bottleneck? Is there a major challenge to using 
 a thread pool of DataXceivers in this situation?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-362) FSEditLog should not writes long and short as UTF8 and should not use ArrayWritable for writing non-array items

2011-09-16 Thread Tsz Wo (Nicholas), SZE (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-362:


Hadoop Flags: [Reviewed]

+1 patch looks good.

I will wait for a day before committing this, to see if there is any comment 
from the others.

 FSEditLog should not writes long and short as UTF8 and should not use 
 ArrayWritable for writing non-array items
 ---

 Key: HDFS-362
 URL: https://issues.apache.org/jira/browse/HDFS-362
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 0.23.0, 0.24.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Uma Maheswara Rao G
 Attachments: HDFS-362.1.patch, HDFS-362.2.patch, HDFS-362.2b.patch, 
 HDFS-362.2c.patch, HDFS-362.2d.patch, HDFS-362.2d.patch, HDFS-362.2e.patch, 
 HDFS-362.patch


 In FSEditLog, 
 - long and short are first converted to String and are further converted to 
 UTF8
 - For some non-array items, it first create an ArrayWritable object to hold 
 all the items and then writes the ArrayWritable object.
 These result creating many intermediate objects which affects Namenode CPU 
 performance and Namenode restart.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2243) DataXceiver per accept seems to be a bottleneck in HBase/YCSB test

2011-09-16 Thread Eric Caspole (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Caspole updated HDFS-2243:
---

Status: Patch Available  (was: Open)

The HDFS-2243-0.23-110909.patch should apply to trunk

 DataXceiver per accept seems to be a bottleneck in HBase/YCSB test
 --

 Key: HDFS-2243
 URL: https://issues.apache.org/jira/browse/HDFS-2243
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
 Environment: Using Fedora 14 on a quad core phenom system
Reporter: Eric Caspole
Priority: Minor
 Fix For: 0.24.0

 Attachments: HDFS-2234-branch-0.20-append.patch, 
 HDFS-2243-0.23-110909.patch, HDFS-2243-0.23-110909.txt, 
 datanode-perf-110808.gif


 I am running the YCSB benchmark against HBase, sometimes against a single 
 node, sometimes against a cluster of 6 systems. As the load increases into 
 thousands of TPS, especially on the single node, I can see that the datanode 
 runs very high system time and seems to be bottlenecked by how fast it can 
 create the threads to handle the new connections in DataXceiverServer.run. By 
 perf top I can see the process spends about 12% of all its time in 
 pthread_create, and in hprof profiles I can see there are tens of thousands 
 of threads created in just a few minutes of test execution.
 Does anyone else observe this bottleneck? Is there a major challenge to using 
 a thread pool of DataXceivers in this situation?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2318) Provide authentication to webhdfs using SPNEGO

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106517#comment-13106517
 ] 

Hudson commented on HDFS-2318:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #914 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/914/])
Revised the entry for HDFS-2318 in CHANGES.txt.
HDFS-2318. Provide authentication to webhdfs using SPNEGO and delegation tokens.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171615
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171611
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetDelegationTokenServlet.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/AuthFilter.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/KerberosUgiAuthenticator.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BlockSizeParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DelegationParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/HttpOpParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserProvider.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java


 Provide authentication to webhdfs using SPNEGO
 --

 Key: HDFS-2318
 URL: https://issues.apache.org/jira/browse/HDFS-2318
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Fix For: 0.20.205.0, 0.24.0

 Attachments: h2318_20110914.patch, h2318_20110914c.patch, 
 h2318_20110914c_0.20s.patch, h2318_20110915.patch, 
 h2318_20110915_0.20s.patch, h2318_20110915b.patch, 
 h2318_20110915b_0.20s.patch, h2318_20110916.patch, h2318_20110916_0.20s.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2243) DataXceiver per accept seems to be a bottleneck in HBase/YCSB test

2011-09-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106545#comment-13106545
 ] 

Hadoop QA commented on HDFS-2243:
-

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12494031/HDFS-2243-0.23-110909.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed these unit tests:
  
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
  org.apache.hadoop.hdfs.TestDfsOverAvroRpc

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/1260//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1260//console

This message is automatically generated.

 DataXceiver per accept seems to be a bottleneck in HBase/YCSB test
 --

 Key: HDFS-2243
 URL: https://issues.apache.org/jira/browse/HDFS-2243
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
 Environment: Using Fedora 14 on a quad core phenom system
Reporter: Eric Caspole
Priority: Minor
 Fix For: 0.24.0

 Attachments: HDFS-2234-branch-0.20-append.patch, 
 HDFS-2243-0.23-110909.patch, HDFS-2243-0.23-110909.txt, 
 datanode-perf-110808.gif


 I am running the YCSB benchmark against HBase, sometimes against a single 
 node, sometimes against a cluster of 6 systems. As the load increases into 
 thousands of TPS, especially on the single node, I can see that the datanode 
 runs very high system time and seems to be bottlenecked by how fast it can 
 create the threads to handle the new connections in DataXceiverServer.run. By 
 perf top I can see the process spends about 12% of all its time in 
 pthread_create, and in hprof profiles I can see there are tens of thousands 
 of threads created in just a few minutes of test execution.
 Does anyone else observe this bottleneck? Is there a major challenge to using 
 a thread pool of DataXceivers in this situation?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly

2011-09-16 Thread Eli Collins (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106562#comment-13106562
 ] 

Eli Collins commented on HDFS-2246:
---

@Arun, no but I think we should respect the 20x guidelines that Owen and Matt 
established, in particular that features go into trunk prior to branch date.

Wrt HDFS-2285 if 22 contains a lot of stuff like this I wouldn't vote +1 for 
the release.

 Shortcut a local client reads to a Datanodes files directly
 ---

 Key: HDFS-2246
 URL: https://issues.apache.org/jira/browse/HDFS-2246
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Sanjay Radia
 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246.20s.1.patch, 
 HDFS-2246.20s.patch, localReadShortcut20-security.2patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2243) DataXceiver per accept seems to be a bottleneck in HBase/YCSB test

2011-09-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106565#comment-13106565
 ] 

Hadoop QA commented on HDFS-2243:
-

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12494802/HDFS-2243-0.23-110909.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed these unit tests:
  org.apache.hadoop.hdfs.TestDfsOverAvroRpc
  
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/1261//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1261//console

This message is automatically generated.

 DataXceiver per accept seems to be a bottleneck in HBase/YCSB test
 --

 Key: HDFS-2243
 URL: https://issues.apache.org/jira/browse/HDFS-2243
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
 Environment: Using Fedora 14 on a quad core phenom system
Reporter: Eric Caspole
Priority: Minor
 Fix For: 0.24.0

 Attachments: HDFS-2234-branch-0.20-append.patch, 
 HDFS-2243-0.23-110909.patch, HDFS-2243-0.23-110909.txt, 
 datanode-perf-110808.gif


 I am running the YCSB benchmark against HBase, sometimes against a single 
 node, sometimes against a cluster of 6 systems. As the load increases into 
 thousands of TPS, especially on the single node, I can see that the datanode 
 runs very high system time and seems to be bottlenecked by how fast it can 
 create the threads to handle the new connections in DataXceiverServer.run. By 
 perf top I can see the process spends about 12% of all its time in 
 pthread_create, and in hprof profiles I can see there are tens of thousands 
 of threads created in just a few minutes of test execution.
 Does anyone else observe this bottleneck? Is there a major challenge to using 
 a thread pool of DataXceivers in this situation?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Eli Collins (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106578#comment-13106578
 ] 

Eli Collins commented on HDFS-2337:
---

+1

 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Aaron T. Myers (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106579#comment-13106579
 ] 

Aaron T. Myers commented on HDFS-2337:
--

Thanks a lot for the review, Eli. I will commit this in an hour or so unless 
there are more comments in the mean time.

 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2294) Download of commons-daemon TAR should not be under target

2011-09-16 Thread Alejandro Abdelnur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HDFS-2294:
-

Attachment: HDFS-2289.patch

When SVN committing the directory 'hadoop-hdfs-project/hadoop-hdfs/downloads' 
should be  svn:ignored

 Download of commons-daemon TAR should not be under target
 -

 Key: HDFS-2294
 URL: https://issues.apache.org/jira/browse/HDFS-2294
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 0.23.0, 0.24.0
Reporter: Alejandro Abdelnur
 Fix For: 0.23.0, 0.24.0

 Attachments: HDFS-2289.patch


 Committed HDFS-2289 downloads commons-daemon TAR in the hadoop-hdfs/target/, 
 earlier patches for HDFS-2289 were using hadoop-hdfs/download/ as the 
 location for the download.
 The motivation not to use the 'target/' directory is that on every clean 
 build the TAR will be downloaded from Apache archives. Using a 'download' 
 directory this happens once per workspace.
 The patch was also adding the 'download/' directory to the .gitignore file 
 (it should also be svn ignored).
 Besides downloading it only once, it allows to do a clean build in 
 disconnected mode.
 IMO, the later is a nice developer capability.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2294) Download of commons-daemon TAR should not be under target

2011-09-16 Thread Alejandro Abdelnur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HDFS-2294:
-

Affects Version/s: 0.24.0
Fix Version/s: 0.24.0
 Assignee: Alejandro Abdelnur

 Download of commons-daemon TAR should not be under target
 -

 Key: HDFS-2294
 URL: https://issues.apache.org/jira/browse/HDFS-2294
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 0.23.0, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 0.23.0, 0.24.0

 Attachments: HDFS-2289.patch


 Committed HDFS-2289 downloads commons-daemon TAR in the hadoop-hdfs/target/, 
 earlier patches for HDFS-2289 were using hadoop-hdfs/download/ as the 
 location for the download.
 The motivation not to use the 'target/' directory is that on every clean 
 build the TAR will be downloaded from Apache archives. Using a 'download' 
 directory this happens once per workspace.
 The patch was also adding the 'download/' directory to the .gitignore file 
 (it should also be svn ignored).
 Besides downloading it only once, it allows to do a clean build in 
 disconnected mode.
 IMO, the later is a nice developer capability.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2294) Download of commons-daemon TAR should not be under target

2011-09-16 Thread Alejandro Abdelnur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HDFS-2294:
-

Status: Patch Available  (was: Open)

 Download of commons-daemon TAR should not be under target
 -

 Key: HDFS-2294
 URL: https://issues.apache.org/jira/browse/HDFS-2294
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 0.23.0, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 0.23.0, 0.24.0

 Attachments: HDFS-2289.patch


 Committed HDFS-2289 downloads commons-daemon TAR in the hadoop-hdfs/target/, 
 earlier patches for HDFS-2289 were using hadoop-hdfs/download/ as the 
 location for the download.
 The motivation not to use the 'target/' directory is that on every clean 
 build the TAR will be downloaded from Apache archives. Using a 'download' 
 directory this happens once per workspace.
 The patch was also adding the 'download/' directory to the .gitignore file 
 (it should also be svn ignored).
 Besides downloading it only once, it allows to do a clean build in 
 disconnected mode.
 IMO, the later is a nice developer capability.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2294) Download of commons-daemon TAR should not be under target

2011-09-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106601#comment-13106601
 ] 

Hadoop QA commented on HDFS-2294:
-

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12494820/HDFS-2289.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1262//console

This message is automatically generated.

 Download of commons-daemon TAR should not be under target
 -

 Key: HDFS-2294
 URL: https://issues.apache.org/jira/browse/HDFS-2294
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 0.23.0, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 0.23.0, 0.24.0

 Attachments: HDFS-2289.patch


 Committed HDFS-2289 downloads commons-daemon TAR in the hadoop-hdfs/target/, 
 earlier patches for HDFS-2289 were using hadoop-hdfs/download/ as the 
 location for the download.
 The motivation not to use the 'target/' directory is that on every clean 
 build the TAR will be downloaded from Apache archives. Using a 'download' 
 directory this happens once per workspace.
 The patch was also adding the 'download/' directory to the .gitignore file 
 (it should also be svn ignored).
 Besides downloading it only once, it allows to do a clean build in 
 disconnected mode.
 IMO, the later is a nice developer capability.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2341) Contribs not building

2011-09-16 Thread Joep Rottinghuis (JIRA) 
Contribs not building
-

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0


Contribs are not getting built.
Snippet from Jenkins:

compile:
   [subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2341) Contribs not building

2011-09-16 Thread Joep Rottinghuis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106663#comment-13106663
 ] 

Joep Rottinghuis commented on HDFS-2341:


See: 
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/84/console

 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Aaron T. Myers (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers updated HDFS-2337:
-

  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

I've just committed this.

 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2341) Contribs not building

2011-09-16 Thread Joep Rottinghuis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated HDFS-2341:
---

Status: Patch Available  (was: Open)

 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: HDFS-2341-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2341) Contribs not building

2011-09-16 Thread Joep Rottinghuis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joep Rottinghuis updated HDFS-2341:
---

Attachment: HDFS-2341-branch-0.22.patch

 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: HDFS-2341-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106681#comment-13106681
 ] 

Hudson commented on HDFS-2337:
--

Integrated in Hadoop-Hdfs-trunk-Commit #977 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/977/])
HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171711
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java


 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106679#comment-13106679
 ] 

Hudson commented on HDFS-2337:
--

Integrated in Hadoop-Common-trunk-Commit #900 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/900/])
HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171711
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java


 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2341) Contribs not building

2011-09-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106682#comment-13106682
 ] 

Hadoop QA commented on HDFS-2341:
-

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12494843/HDFS-2341-branch-0.22.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1263//console

This message is automatically generated.

 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: HDFS-2341-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2337) DFSClient shouldn't keep multiple RPC proxy references

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106700#comment-13106700
 ] 

Hudson commented on HDFS-2337:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #915 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/915/])
HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171711
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java


 DFSClient shouldn't keep multiple RPC proxy references
 --

 Key: HDFS-2337
 URL: https://issues.apache.org/jira/browse/HDFS-2337
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: hdfs-2337.patch


 With the commit of HADOOP-7635, {{RetryInvocationHandler}} objects will clean 
 up the underlying {{InvocationHandler}} objects they reference. We should 
 change {{DFSClient}} to take advantage of this fact.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2342) TestSleepJob and TestHdfsProxy broken after HDFS-2284

2011-09-16 Thread Kihwal Lee (JIRA) 
TestSleepJob and TestHdfsProxy broken after HDFS-2284
-

 Key: HDFS-2342
 URL: https://issues.apache.org/jira/browse/HDFS-2342
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: contrib/hdfsproxy, test
Affects Versions: 0.20.205.0
Reporter: Kihwal Lee
 Fix For: 0.20.205.0


After HDFS-2284, TestSleepJob and TestHdfsProxy are failing.
The both work in rev 1167444 and fail in rev 1167663.
It will be great if they can be fixed for 205.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2342) TestSleepJob and TestHdfsProxy broken after HDFS-2284

2011-09-16 Thread Kihwal Lee (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-2342:
-

Fix Version/s: 0.20.206.0

 TestSleepJob and TestHdfsProxy broken after HDFS-2284
 -

 Key: HDFS-2342
 URL: https://issues.apache.org/jira/browse/HDFS-2342
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: contrib/hdfsproxy, test
Affects Versions: 0.20.205.0
Reporter: Kihwal Lee
 Fix For: 0.20.205.0, 0.20.206.0


 After HDFS-2284, TestSleepJob and TestHdfsProxy are failing.
 The both work in rev 1167444 and fail in rev 1167663.
 It will be great if they can be fixed for 205.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2343) Make hdfs use same version of avro as HBase

2011-09-16 Thread Joep Rottinghuis (JIRA) 
Make hdfs use same version of avro as HBase
---

 Key: HDFS-2343
 URL: https://issues.apache.org/jira/browse/HDFS-2343
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0


HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
When building HBase on top of hadoop, this should be consistent.
Moreover, this should be consistent between common, hdfs, and mapreduce.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2306) NameNode web UI should show information about recent checkpoints

2011-09-16 Thread Todd Lipcon (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106883#comment-13106883
 ] 

Todd Lipcon commented on HDFS-2306:
---

- Not crazy about the new config name... dfs.namenode.checkpoint.events is 
kind of vague. Perhaps dfs.namenode.checkpoint.history.retained ?
- missing license on CheckpointEvent.java
- does CheckpointEvent need to be public in order to work as part of JMX? I'm 
not a JMX pro. If public, it should have audience/stability annotations
- we may want to expose getSyncTxId() instead of getLastWrittenTxId()
- typo insce the last checkpoint in NameNodeMXBean
- getRecentCheckpointEvents doesn't return a copy of the list, so the web page 
may throw ConcurrentModificationException. Perhaps change 
recentCheckpointEvents to be a CopyOnWriteArrayList? Or return a copy?

I'm still unclear on what the fields are inside CheckpointEvent... maybe we can 
sync up offline to discuss.


 NameNode web UI should show information about recent checkpoints
 

 Key: HDFS-2306
 URL: https://issues.apache.org/jira/browse/HDFS-2306
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 0.24.0

 Attachments: checkpoint-history.1.png, checkpoint-history.png, 
 hdfs-2306.0.patch, hdfs-2306.1.patch


 It would be nice if the NN web UI showed the 2NN address, timestamp, number 
 of edits, etc. of the last few checkpoints.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2341) Contribs not building

2011-09-16 Thread Konstantin Shvachko (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106905#comment-13106905
 ] 

Konstantin Shvachko commented on HDFS-2341:
---

I'll commit this and see if how Jenkins reacts.

 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: HDFS-2341-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1973) HA: HDFS clients must handle namenode failover and switch over to the new active namenode.

2011-09-16 Thread Aaron T. Myers (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers updated HDFS-1973:
-

Attachment: HDFS-1973-HDFS-1623.patch

Here's an updated patch which builds upon the previous patch and addresses all 
of the issues I mentioned above.

The key differences between this patch and the previous:

# Take advantage of the changes in HADOOP-7635 and HDFS-2337 to enable cleanup 
of RPC connection resources.
# Add the @Idempotent annotation to more methods of {{ClientProtocol}}.
# Add some appropriate synchronization to {{ConfiguredFailoverProxyProvider}}.

Please review this patch for potential inclusion in the HDFS-1623 branch.

 HA: HDFS clients must handle namenode failover and switch over to the new 
 active namenode.
 --

 Key: HDFS-1973
 URL: https://issues.apache.org/jira/browse/HDFS-1973
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Suresh Srinivas
Assignee: Aaron T. Myers
 Attachments: HDFS-1973-HDFS-1623.patch, hdfs-1973.0.patch


 During failover, a client must detect the current active namenode failure and 
 switch over to the new active namenode. The switch over might make use of IP 
 failover or some thing more elaborate such as zookeeper to discover the new 
 active.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2341) Contribs not building

2011-09-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106921#comment-13106921
 ] 

Hudson commented on HDFS-2341:
--

Integrated in Hadoop-Hdfs-22-branch #85 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-22-branch/85/])
HDFS-2341. Contribs not building. Contributed by Joep Rottinghuis.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171857
Files : 
* /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.22/hdfs/build.xml
* /hadoop/common/branches/branch-0.22/hdfs/src/contrib/build.xml


 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: HDFS-2341-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1835) DataNode.setNewStorageID pulls entropy from /dev/random

2011-09-16 Thread John Carrino (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106993#comment-13106993
 ] 

John Carrino commented on HDFS-1835:


If you just 'cat /dev/urandom  /dev/null' it will eat entropy.  urandom 
prefers entropy, but doesn't require it.  Even if you are using urandom, you 
can push the entropy low enough that if someone then uses /dev/random, it may 
block.

 DataNode.setNewStorageID pulls entropy from /dev/random
 ---

 Key: HDFS-1835
 URL: https://issues.apache.org/jira/browse/HDFS-1835
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.20.2
Reporter: John Carrino
Assignee: John Carrino
 Fix For: 0.23.0

 Attachments: DataNode.patch, hdfs-1835.txt

   Original Estimate: 10m
  Remaining Estimate: 10m

 DataNode.setNewStorageID uses SecureRandom.getInstance(SHA1PRNG) which 
 always pulls fresh entropy.
 It wouldn't be so bad if this were only the 120 bits needed by sha1, but the 
 default impl of SecureRandom actually uses a BufferedInputStream around 
 /dev/random and pulls 1024 bits of entropy for this one call.
 If you are on a system without much entropy coming in, this call can block 
 and block others.
 Can we just change this to use new 
 SecureRandom().nextInt(Integer.MAX_VALUE) instead?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2341) Contribs not building

2011-09-16 Thread Joep Rottinghuis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107034#comment-13107034
 ] 

Joep Rottinghuis commented on HDFS-2341:


Mapred one was clean.
Seems that the hdfs one is failing on clean target. Will look into that and 
provide fix.

 Contribs not building
 -

 Key: HDFS-2341
 URL: https://issues.apache.org/jira/browse/HDFS-2341
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.22.0
Reporter: Joep Rottinghuis
Assignee: Joep Rottinghuis
Priority: Blocker
 Fix For: 0.22.0

 Attachments: HDFS-2341-branch-0.22.patch


 Contribs are not getting built.
 Snippet from Jenkins:
 compile:
[subant] No sub-builds to iterate on

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira