[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5594: Attachment: HDFS-5594.7.patch Thanks, Nicholas. I'm uploading patch version 7. I plan to commit this version to the HDFS-4685 branch tomorrow unless there is more feedback by then. bq. In FileSystem, should modifyAclEntries be called as addAclEntries? This method actually merges the requested modifications into the file's existing ACL. The aclSpec can contain a mix of adding new ACL entries and modifying the permissions associated with existing ACL entries. If the file has existing ACL entries that are not mentioned in the aclSpec, then those entries are retained. I think {{modifyAclEntries}} is the appropriate name, but I also think my JavaDocs didn't explain this clearly enough. I've tried to improve on this in the new patch. bq. Typo in Acl: associatd Thanks for catching this. I fixed it. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch, HDFS-5594.7.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844035#comment-13844035 ] Hadoop QA commented on HDFS-5594: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12618000/HDFS-5594.6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5687//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5687//console This message is automatically generated. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5594: - Hadoop Flags: Reviewed +1 patch looks good. Minor question/typo: - In FileSystem, should modifyAclEntries be called as addAclEntries? Both "add" and "remove" are modifying the entries but "modify" here means "add". - Typo in Acl: associatd > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5594: Attachment: HDFS-5594.6.patch bq. Since you are touching FileSystem.java can clean following unused imports Sure thing. Here is patch version 6 to make that change. bq. One more thing, (general), why cant we check of trailing space in modified lines using Hadoop QA? This is done in bookkeeper's QA. I'm not sure if this is an enforced standard for the project, although it is something I strive to do in my patches. I suggest filing a separate jira to do this, and if anyone objects, then they can discuss there. It's definitely feasible to add a check for this in test-patch.sh. It would need to be smart enough to detect trailing whitespace only on new code, not diff context from existing code. i.e. something like: {code} grep '^\+.*\s$' {code} > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843999#comment-13843999 ] Liang Xie commented on HDFS-5637: - Good catch, V2 should address comments from Colin and Junping, seems i am too lazy to do a grep before make a patch... For refactoring, it doesn't repeat more than twice and the code section is pretty short, to me it still ok for leaving it unchanged:) > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5637-v2.txt, HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
[jira] [Updated] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5637: Attachment: HDFS-5637-v2.txt > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5637-v2.txt, HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) > at > org.apache.hadoop.ipc.RemoteException.unwrap
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843980#comment-13843980 ] Hadoop QA commented on HDFS-4201: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617973/trunk-4201_v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated -12 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5685//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5685//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5685//console This message is automatically generated. > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch, > trunk-4201_v3.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843948#comment-13843948 ] Hadoop QA commented on HDFS-1605: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617846/HDFS-1605.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5684//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5684//console This message is automatically generated. > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt, > HDFS-1605.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843932#comment-13843932 ] Yongjun Zhang commented on HDFS-4983: - Thanks a lot Andrew! > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.3.0 > > Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch, > HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch, > HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843907#comment-13843907 ] Junping Du commented on HDFS-5637: -- Thanks [~xieliang007] for the patch! Agree with Colin that handling exception in getBlockReader() in fetchBlockByteRange should be the same. In addition, I would suggest to refactor the exception handling code here to be a method and get shared in places calling getBlockReader(). How it sounds? > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstruct
[jira] [Updated] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HDFS-4201: -- Attachment: trunk-4201_v3.patch Attached v3 that isolated the changes as suggested. > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch, > trunk-4201_v3.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4960) Unnecessary .meta seeks even when skip checksum is true
[ https://issues.apache.org/jira/browse/HDFS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843883#comment-13843883 ] Hadoop QA commented on HDFS-4960: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12591131/4960-trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated -14 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.hdfs.web.TestWebHDFS {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5683//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5683//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5683//console This message is automatically generated. > Unnecessary .meta seeks even when skip checksum is true > --- > > Key: HDFS-4960 > URL: https://issues.apache.org/jira/browse/HDFS-4960 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Varun Sharma >Assignee: Varun Sharma > Attachments: 4960-branch2.patch, 4960-trunk.patch > > > While attempting to benchmark an HBase + Hadoop 2.0 setup on SSDs, we found > unnecessary seeks into .meta files, each seek was a 7 byte read at the head > of the file - this attempts to validate the version #. Since the client is > requesting no-checksum, we should not be needing to touch the .meta file at > all. > Since the purpose of skip checksum is to also avoid the performance penalty > of the extra seek, we should not be seeking into .meta if skip checksum is > true -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843873#comment-13843873 ] Hudson commented on HDFS-4983: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4857 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4857/]) Move HDFS-4983 to 2.3 in CHANGES.txt (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1549747) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.3.0 > > Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch, > HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch, > HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-4983: -- Fix Version/s: (was: 2.4.0) 2.3.0 Committed to branch-2.3, fixed CHANGES.txt in branch-2 and trunk. > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.3.0 > > Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch, > HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch, > HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843867#comment-13843867 ] Colin Patrick McCabe commented on HDFS-4201: thanks, Jimmy. I am +1 once this is addressed. > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-1605: Status: Patch Available (was: Open) > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt, > HDFS-1605.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843859#comment-13843859 ] Hadoop QA commented on HDFS-5647: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617937/HDFS-5647.000.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5681//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5681//console This message is automatically generated. > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5647.000.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843836#comment-13843836 ] Andrew Wang commented on HDFS-4983: --- +1 looks good to me, thanks Yongjun. Typically we don't put the CHANGES.txt in the generated patch, but that's easy for me to fixup. Will commit to branch-2.3 shortly and shuffle CHANGES.txt accordingly. > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.4.0 > > Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch, > HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch, > HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843820#comment-13843820 ] Vinay commented on HDFS-5594: - Hi Chris, latest patch looks great. +1 I have some small concern, Not exactly about your patch. Since you are touching FileSystem.java can clean following unused imports {code}import java.util.Arrays; import java.util.Collections; import java.util.concurrent.atomic.AtomicInteger;{code} One more thing, (general), why cant we check of trailing space in modified lines using Hadoop QA? This is done in bookkeeper's QA. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4710) SCR should honor dfs.client.read.shortcircuit.buffer.size even when checksums are off
[ https://issues.apache.org/jira/browse/HDFS-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843780#comment-13843780 ] Hadoop QA commented on HDFS-4710: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12589940/HDFS-4710.002.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5682//console This message is automatically generated. > SCR should honor dfs.client.read.shortcircuit.buffer.size even when checksums > are off > - > > Key: HDFS-4710 > URL: https://issues.apache.org/jira/browse/HDFS-4710 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.0.4-alpha > Environment: Centos (EC2) + short-circuit reads on >Reporter: Gopal V >Assignee: Colin Patrick McCabe >Priority: Minor > Labels: perfomance > Attachments: HDFS-4710.001.patch, HDFS-4710.002.patch > > > When short-circuit reads are on, HDFS client slows down when checksums are > turned off. > With checksums on, the query takes 45.341 seconds and with it turned off, it > takes 56.345 seconds. This is slower than the speeds observed when > short-circuiting is turned off. > The issue seems to be that FSDataInputStream.readByte() calls are directly > transferred to the disk fd when the checksums are turned off. > Even though all the columns are integers, the data being read will be read > via DataInputStream which does > {code} > public final int readInt() throws IOException { > int ch1 = in.read(); > int ch2 = in.read(); > int ch3 = in.read(); > int ch4 = in.read(); > {code} > To confirm, an strace of the Yarn container shows > {code} > 26690 read(154, "B", 1) = 1 > 26690 read(154, "\250", 1) = 1 > 26690 read(154, ".", 1) = 1 > 26690 read(154, "\24", 1) = 1 > {code} > To emulate this without the entirety of Hive code, I have written a simpler > test app > https://github.com/t3rmin4t0r/shortcircuit-reader > The jar will read a file in -bs sized buffers. Running it with 1 byte > blocks gives similar results to the Hive test run. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[ https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843767#comment-13843767 ] Andrew Wang commented on HDFS-5634: --- This is good stuff, thanks Colin. I have a few review comments: BRL-related: * Do we mean to setCachingStrategy in DFSInputStream#getBlockReader? Also, I get that there are a zillion parameters for the BRL constructor, but builders are for when there are optional arguments. Here, it looks like we want to set all of them. * We have both {{verifyChecksum}} and {{skipChecksum}} right now. Let's get rid of one, seems error-prone to be flipping booleans. * The {{skipChecksum || mlocked.get()}} idiom is used in a few places, maybe should be a {{shouldSkipChecksum()}} method? * IIUC, {{fillDataBuf}} fills the bounce buffer, and {{drainBounceBuffer}} empties it. Rename {{fillDataBuf}} to {{fillBounceBuffer}} for parity? * BlockReaderLocal:500, trailing whitespace * I'm wondering what happens in the bounce buffer read paths when readahead is turned off. It looks like they use readahead to determine how much to read, regardless of the bytes needed, so what if it's zero? * For the slow lane, {{fillDataBuf}} doesn't actually fill the returned buf, so when we hit the EOF and break, it looks like we make the user read again to flush out the bounce buffer. Can we save this? * {{fillDataBuf}} doesn't fill just the data buf, it also fills the checksum buf and verifies checksums via {{fillBuffer}}. Would be nice to javadoc this. * I noticed there are two readahead config options too, dfs.client.cache.readahead and dfs.datanode.readahead.bytes. Seems like we should try to emulate the same behavior as remote reads which (according to hdfs-default.xml) use the DN setting, and override with the client setting. Right now it's just using the DN readahead in BRL, so the test that sets client readahead to 0 isn't doing much. * I don't quite understand why we check {{needed > maxReadahead...}} for the fast lane. Once we're checksum aligned via draining the bounce buffer, can't we just stay in the fast lane? Seems like the slow vs. fast lane determination should be based on read alignment, not bytes left. Little stuff: * It's a little weird to me that the readahead chunks is min'd with the buffer size (default 1MB). I get why (memory consumption), but this linkage should be documented somewhere. * DirectBufferPool, would it be better to use a Deque's stack operations rather than a Queue? Might give better cache locality to do LIFO rather than FIFO. * TestEnhancedByteBufferAccess has an import only change * Kinda unrelated, but should the "dfs.client.read.shortcircuit.*" keys be in hdfs-default.xml? I also noticed that "dfs.client.cache.readahead" says "this setting causes the datanode to..." so the readahead documentation might need to be updated too. > allow BlockReaderLocal to switch between checksumming and not > - > > Key: HDFS-5634 > URL: https://issues.apache.org/jira/browse/HDFS-5634 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch > > > BlockReaderLocal should be able to switch between checksumming and > non-checksumming, so that when we get notifications that something is mlocked > (see HDFS-5182), we can avoid checksumming when reading from that block. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Attachment: HDFS-5647.000.patch > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5647.000.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Description: HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features introduced in HDFS-5284. The current code separates the INode feature of INodeFile and INodeDirectory into two different class hierarchies. This hinders the implementation of ACL since ACL is a concept that applies to both INodeFile and INodeDirectory. This jira proposes to merge the two class hierarchies (i.e., INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation of ACLs. > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5647.000.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Status: Patch Available (was: Open) > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5647.000.patch > > > HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features > introduced in HDFS-5284. The current code separates the INode feature of > INodeFile and INodeDirectory into two different class hierarchies. This > hinders the implementation of ACL since ACL is a concept that applies to both > INodeFile and INodeDirectory. > This jira proposes to merge the two class hierarchies (i.e., > INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation > of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-5647) Merge
Haohui Mai created HDFS-5647: Summary: Merge Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Assigned] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5647: Assignee: Haohui Mai > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Summary: Merge INodeDirectory.Feature and INodeFile.Feature (was: Merge) > Merge INodeDirectory.Feature and INodeFile.Feature > -- > > Key: HDFS-5647 > URL: https://issues.apache.org/jira/browse/HDFS-5647 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based quota management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843691#comment-13843691 ] Andrew Wang commented on HDFS-5431: --- I ran the OEV test locally, passes once you do the copy: {noformat} --- T E S T S --- Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.455 sec - in org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Results : Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 30.613s [INFO] Finished at: Mon Dec 09 15:31:47 PST 2013 [INFO] Final Memory: 32M/560M [INFO] {noformat} > support cachepool-based quota management in path-based caching > -- > > Key: HDFS-5431 > URL: https://issues.apache.org/jira/browse/HDFS-5431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch > > > We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based quota management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843686#comment-13843686 ] Hadoop QA commented on HDFS-5431: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617896/hdfs-5431-2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5678//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5678//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5678//console This message is automatically generated. > support cachepool-based quota management in path-based caching > -- > > Key: HDFS-5431 > URL: https://issues.apache.org/jira/browse/HDFS-5431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch > > > We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843685#comment-13843685 ] Hadoop QA commented on HDFS-5594: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617918/HDFS-5594.5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5680//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5680//console This message is automatically generated. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5594: Attachment: HDFS-5594.5.patch Ah, I hadn't checked for trailing whitespace on the comment lines. Here is patch version 5 to clean that up. I didn't see any remaining imports of the commons-lang stuff, so patch version 5 doesn't change anything related to that. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch, HDFS-5594.5.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843622#comment-13843622 ] Jimmy Xiang commented on HDFS-4201: --- Sure, I will do as suggested so that we can minimize the changes. Thanks. > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843620#comment-13843620 ] Hadoop QA commented on HDFS-5594: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617908/HDFS-5594.4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5679//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5679//console This message is automatically generated. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843619#comment-13843619 ] Colin Patrick McCabe commented on HDFS-4201: What about this paradigm? {code} boolean success = false; try { ... success = true } finally { if (!success) { cleanup } } {code} we use this paradigm many places in hdfs to avoid having to enumerate all possible exceptions. > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5629) Support HTTPS in JournalNode and SecondaryNameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843612#comment-13843612 ] Hudson commented on HDFS-5629: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4855/]) HDFS-5629. Support HTTPS in JournalNode and SecondaryNameNode. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1549692) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/QJournalProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java > Support HTTPS in JournalNode and SecondaryNameNode > -- > > Key: HDFS-5629 > URL: https://issues.apache.org/jira/browse/HDFS-5629 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0 > > Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, > HDFS-5629.002.patch, HDFS-5629.003.patch > > > Currently JournalNode has only HTTP support only. This jira tracks the effort > to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5639) rpc scheduler abstraction
[ https://issues.apache.org/jira/browse/HDFS-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843606#comment-13843606 ] Chris Li commented on HDFS-5639: Something like this will be needed down the road if HADOOP-9640 is adopted; I'll open separate jiras for these enhancements when we're ready. > rpc scheduler abstraction > - > > Key: HDFS-5639 > URL: https://issues.apache.org/jira/browse/HDFS-5639 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma > Attachments: HDFS-5639-2.patch, HDFS-5639.patch > > > We have run into various issues in namenode and hbase w.r.t. rpc handling in > multi-tenant clusters. The examples are > https://issues.apache.org/jira/i#browse/HADOOP-9640 > https://issues.apache.org/jira/i#browse/HBASE-8836 > There are different ideas on how to prioritize rpc requests. It could be > based on user id, or whether it is read request or write request, or it could > use specific rule like datanode's RPC is more important than client RPC. > We want to enable people to implement and experiiment different rpc > schedulers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843599#comment-13843599 ] Haohui Mai commented on HDFS-5594: -- btw, there're a couple whitespace in the new patch, all of which reside in the comments (e.g., line 24, 38, 52). The import statements below can be deleted: {code} +import org.apache.commons.lang.builder.EqualsBuilder; +import org.apache.commons.lang.builder.HashCodeBuilder; {code} +1 on the new patch. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5629) Support HTTPS in JournalNode and SecondaryNameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5629: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 for the 003 patch. I've committed this to trunk. > Support HTTPS in JournalNode and SecondaryNameNode > -- > > Key: HDFS-5629 > URL: https://issues.apache.org/jira/browse/HDFS-5629 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0 > > Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, > HDFS-5629.002.patch, HDFS-5629.003.patch > > > Currently JournalNode has only HTTP support only. This jira tracks the effort > to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5594: Attachment: HDFS-5594.4.patch Here is version 4 of the patch with following changes: # {{equals}} and {{hashCode}} methods have switched from Commons Lang to Guava. # {{Acl#entries}} is converted from {{Set}} to {{List}}. Let's go with that, and the NameNode implementation can still enforce the unique constraint that I mentioned. # I implemented {{AclEntry#compareTo}} and sorted the entries on the way out of {{Acl#Builder}}. I added another test for this. Haohui, the only trailing whitespace I saw was in the diff context for existing code, not the new code. {{ViewFileSystem}} has a lot of trailing whitepsace right now, and I'd prefer not to roll a full cleanup into this patch. If you see trailing whitespace elsewhere in the new code, please let me know. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch, > HDFS-5594.4.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5629) Support HTTPS in JournalNode and Secondary NameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5629: Summary: Support HTTPS in JournalNode and Secondary NameNode (was: Support HTTPS in JournalNode) > Support HTTPS in JournalNode and Secondary NameNode > --- > > Key: HDFS-5629 > URL: https://issues.apache.org/jira/browse/HDFS-5629 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, > HDFS-5629.002.patch, HDFS-5629.003.patch > > > Currently JournalNode has only HTTP support only. This jira tracks the effort > to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5629) Support HTTPS in JournalNode and SecondaryNameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5629: Summary: Support HTTPS in JournalNode and SecondaryNameNode (was: Support HTTPS in JournalNode and Secondary NameNode) > Support HTTPS in JournalNode and SecondaryNameNode > -- > > Key: HDFS-5629 > URL: https://issues.apache.org/jira/browse/HDFS-5629 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, > HDFS-5629.002.patch, HDFS-5629.003.patch > > > Currently JournalNode has only HTTP support only. This jira tracks the effort > to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Assigned] (HDFS-5596) DistributedFileSystem: implement getAcls and setAcl.
[ https://issues.apache.org/jira/browse/HDFS-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5596: Assignee: Haohui Mai > DistributedFileSystem: implement getAcls and setAcl. > > > Key: HDFS-5596 > URL: https://issues.apache.org/jira/browse/HDFS-5596 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Haohui Mai > > Implement and test {{getAcls}} and {{setAcl}} in {{DistributedFileSystem}}. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based quota management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843560#comment-13843560 ] Andrew Wang commented on HDFS-5431: --- Thanks Colin, new patch. Rundown of changes: * I re-did the FSImager serialization to use a flag int. I also adjusted CacheDirective to use an int instead of byte, and moved all that code to FSImageSerialization and cleaned it up a bit. I think it's better not to have these methods exposed in a public class. * Removed reservation and weight, le sigh * Renamed quota to limit, hope that's reasonable * Added pool stat for "bytesOverlimit". I guess this could be in the cache directive too, but logically it's more of a pool setting. * Used parseShort I still like having a hard fail when you're at quota. This is how we force someone to go look at their pool stats and fix things. Accepting further directives in this situation feels like a silent failure, since the user might go off their merry way and only much later realize their data wasn't cached. I imagine monitoring tools will still be watching the statistics and throwing up their own errors too. It might also be worth noting that DN liveness won't affect limits since file sizes are kept on the NN, but it will be affected by adding more files to a directory or writing more data to a file. I agree that we could still use more debugging-type information here. If a user has to go to the logs in an expected scenario, we've failed. > support cachepool-based quota management in path-based caching > -- > > Key: HDFS-5431 > URL: https://issues.apache.org/jira/browse/HDFS-5431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch > > > We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5629) Support HTTPS in JournalNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843555#comment-13843555 ] Hadoop QA commented on HDFS-5629: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617527/HDFS-5629.003.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5677//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5677//console This message is automatically generated. > Support HTTPS in JournalNode > > > Key: HDFS-5629 > URL: https://issues.apache.org/jira/browse/HDFS-5629 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, > HDFS-5629.002.patch, HDFS-5629.003.patch > > > Currently JournalNode has only HTTP support only. This jira tracks the effort > to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843549#comment-13843549 ] Arpit Agarwal commented on HDFS-2832: - {quote} I bring them up again because, 4 months later, I was wondering if you had any thoughts on potential solutions that could be added to the doc. It's fine if automatic migration, open files, more elaborate resource management, and additional storage types are all not in immediate scope, but I assume we'll want them in the future. {quote} Andrew, automatic migration is not in scope for our design. wrt open files can you describe a specific use case you think we should be handling that we have not described? Maybe it will help me understand your concern better. If you are concerned about reclaiming capacity for in use blocks, that is analogous to asking "If a process keeps a long-lived handle to a file what will the operating system do to reclaim disk space used by the file?" and the answer is the same - nothing. I don't want anyone reading your comments to get a false impression that the feature is incompatible with SCR. {quote} Well, CDH supports rolling upgrades in some situations. ATM is working on metadata upgrade with HA enabled (HDFS-5138) and I've seen some recent JIRAs related to rolling upgrade (HDFS-5535), so it seems like a reasonable question. At least at the protobuf level, everything so far looks compatible, so I thought it might work as long as the handler code is compatible too. {quote} I am not familiar with how CDH does rolling upgrades so I cannot tell you whether it will work. You recently bumped the layout version for caching so you might recall that HDFS layout version checks prevent a DN registering with an NN with a mismatched version. To my knowledge HDFS-5535 will not fix this limitation either. That said, we have retained wire-protocol compatibility. {quote} Do you forsee heartbeats and block reports always being combined in realistic scenarios? Or are there reasons to split it? Is there any additional overhead from splitting? Can we save any complexity by not supporting split reports? I see this on the test matrix. {quote} I thought I answered it, maybe if you describe your concerns I can give you a better answer. When the test plan says 'split' I meant splitting the reports across multiple requests. Reports will always be split by storage but we are not splitting them across multiple messages for now. What kind of overhead are you thinking of? {quote} b1. Have you put any thought about metrics and tooling to help users and admins debug their quota usage and issues with migrating files to certain storage types? {quote} We'll include it in the next design rev as we start phase 2. > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf, > 20131125-HeterogeneousStorage-TestPlan.pdf, > 20131125-HeterogeneousStorage.pdf, > 20131202-HeterogeneousStorage-TestPlan.pdf, > 20131203-HeterogeneousStorage-TestPlan.pdf, H2832_20131107.patch, > editsStored, h2832_20131023.patch, h2832_20131023b.patch, > h2832_20131025.patch, h2832_20131028.patch, h2832_20131028b.patch, > h2832_20131029.patch, h2832_20131103.patch, h2832_20131104.patch, > h2832_20131105.patch, h2832_20131107b.patch, h2832_20131108.patch, > h2832_20131110.patch, h2832_20131110b.patch, h2832_2013.patch, > h2832_20131112.patch, h2832_20131112b.patch, h2832_20131114.patch, > h2832_20131118.patch, h2832_20131119.patch, h2832_20131119b.patch, > h2832_20131121.patch, h2832_20131122.patch, h2832_20131122b.patch, > h2832_20131123.patch, h2832_20131124.patch, h2832_20131202.patch, > h2832_20131203.patch > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843550#comment-13843550 ] Colin Patrick McCabe commented on HDFS-5637: is there any reason why you didn't change it in fetchBlockByteRange as well? > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.ipc.RemoteException.i
[jira] [Updated] (HDFS-5431) support cachepool-based quota management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-5431: -- Attachment: hdfs-5431-2.patch > support cachepool-based quota management in path-based caching > -- > > Key: HDFS-5431 > URL: https://issues.apache.org/jira/browse/HDFS-5431 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Andrew Wang > Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch > > > We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843547#comment-13843547 ] stack commented on HDFS-5274: - This facility is excellent. A few notes on the patch. Long line and formatting seems off here: {code} + if (header.hasTraceInfo() && header.getTraceInfo() != null && header.getTraceInfo().hasTraceId()) { +String traceDescription = rpcRequest.toString(); + +// If the incoming RPC included tracing info, always continue the trace + TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(), + header.getTraceInfo().getParentId()); + traceSpan = Trace.startSpan(traceDescription, parentSpan).detach(); +} {code} This is a javadoc fix? - * @param buf - SASL wrapped request of one or more RPCs + * @param inBuf - SASL wrapped request of one or more RPCs Are you breaking your pattern of putting trace string building behind a check that tracing is enabled in the below? + TraceScope traceSpan = Trace.startSpan("Call " + method.getName() + " to " + + remoteId, CLIENT_TRACE_SAMPLER); nit: declare and define all in the one line rather than do initialization in constructor: +receivers = new HashSet(); Any reason we take config on construction and in init for SpanReceiverHost? SpanReceiverHost is on only when trace is enabled, right? If so, say so in class comment. Has to be a shutdown hook? ShutdownHookManager.get().addShutdownHook ? This is fine unless we envision someone having to override it which I suppose should never happen for an optionally enabled, rare, trace function? HTraceConfiguration is for testing only? Should be @visiblefortesting only or a comment at least? nit: Do Classname.simple() here instead? +TraceScope scope = Trace.startSpan(DFSInputStream.class.getSimpleName()); Should there be defines for a few of these? "DFSInputStream.close" seems fine... only used once DFSInputStream.read? Is there something off in the formatting here? - - private static class Packet { -private static final long HEART_BEAT_SEQNO = -1L; -long seqno; // sequencenumber of buffer in block -final long offsetInBlock; // offset in block -boolean syncBlock; // this packet forces the current block to disk -int numChunks; // number of chunks currently in packet -final int maxChunks; // max chunks in packet + private Span traceSpan = null; + +private static class Packet { + private static final long HEART_BEAT_SEQNO = -1L; + long seqno; // sequencenumber of buffer in block + final long offsetInBlock; // offset in block + boolean syncBlock; // this packet forces the current block to disk + int numChunks; // number of chunks currently in packet + final int maxChunks; // max chunks in packet Or is it just git cleaning whitespace? Gratuitous change? -ClientOperationHeaderProto header = +ClientOperationHeaderProto.Builder builder = ClientOperationHeaderProto.newBuilder() .setBaseHeader(buildBaseHeader(blk, blockToken)) -.setClientName(client) -.build(); -return header; +.setClientName(client); +return builder.build(); Is an annotation of 'J' only intentional? +if (traceSpan != null) { + traceSpan.addTimelineAnnotation("J"); Put inside a isTracing check? +Trace.addTimelineAnnotation("Connecting to downstream " + mirrorNode); ditto: +Trace.addTimelineAnnotation("Waiting for connect ack from downstream"); > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-4983: Attachment: HDFS-4983-branch2.3.001.patch Add modified patch for branch2.3. Thanks. > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.4.0 > > Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch, > HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch, > HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-5646) Exceptions during HDFS failover
Nikhil Mulley created HDFS-5646: --- Summary: Exceptions during HDFS failover Key: HDFS-5646 URL: https://issues.apache.org/jira/browse/HDFS-5646 Project: Hadoop HDFS Issue Type: Bug Components: ha Reporter: Nikhil Mulley Hi, In our HDFS HA, I see the following excpetions when I try to failback. I have an auto failover mechanism enabled. Although the failback operation succeeds, the exceptions and the return status of 255 tend to worry me (because I cannot script this if I needed to) Please let me know if this is anything that is known and easily resolvable. I am using Cloudera Hadoop 4.4.0, if that helps.Please let me know if I need to open this ticket with CDH Jira, instead. Thanks. sudo -u hdfs hdfs haadmin -failover nn2 nn1 Operation failed: Unable to become active. Service became unhealthy while trying to failover. at org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:652) at org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:58) at org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:591) at org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:588) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:588) at org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94) at org.apache.hadoop.ha.protocolPB.ZKFCProtocolServerSideTranslatorPB.gracefulFailover(ZKFCProtocolServerSideTranslatorPB.java:61) at org.apache.hadoop.ha.proto.ZKFCProtocolProtos$ZKFCProtocolService$2.callBlockingMethod(ZKFCProtocolProtos.java:1351) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1751) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1747) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1745) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843457#comment-13843457 ] Haohui Mai commented on HDFS-5594: -- bq. By including the perms in AclEntry#equals and AclEntry#hashCode, I haven't achieved that. I'm thinking of changing the set to something like Map to better represent the unique constraint. Let me know if you agree, and I'll make the change. I'm wondering whether a compact representation, such as an arraylist might be better fit. This is because that there're relatively few ACLs, and checking the permissions is the dominant operation. Maybe it is a good idea is to spend the effort in addAcl() / removeAcl() to maintain the optimal orders to check the permissions efficiently. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843458#comment-13843458 ] Jimmy Xiang commented on HDFS-4201: --- That's another solution I considered. With try+finally, we need to catch all known and unknown exceptions thrown by initBlockPool, then re-throw it, which may not look very good. Is there any known initialization patch change coming soon? > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843445#comment-13843445 ] Chris Nauroth commented on HDFS-5594: - Thanks for taking a look, Haohui. bq. Why Acl#entries is a set? I'm wondering whether it should be ordered. I chose set rather than something else like list, because the entries are unique. However, looking at this again, I think I've got it wrong. The entries ought to be unique on scope+type+name and the ACL ought to disallow associating that key with different permission settings. By including the perms in {{AclEntry#equals}} and {{AclEntry#hashCode}}, I haven't achieved that. I'm thinking of changing the set to something like {{Map}} to better represent the unique constraint. Let me know if you agree, and I'll make the change. The natural ordering for entries within an ACL is: 1) owner, 2) all named user entries (internal ordering undefined), 3) owning group, 4) all named group entries (internal ordering undefined), 5) other. This is the ordering that will be required when we implement the enforcement logic. I was planning on putting that ordering logic on the NameNode side, but I could also do it here on the model side in a {{AclEntry#Key#compareTo}} method if you prefer. I'll include that version in the next patch. I can switch to the Guava equivalents for {{equals}} and {{hashCode}}. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843436#comment-13843436 ] Haohui Mai commented on HDFS-5594: -- BTW, it seems there're trailing whitespace in the patch. It would be nice if you can fix them when you update the patch. > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[ https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843417#comment-13843417 ] Haohui Mai commented on HDFS-5594: -- The patch looks good to me overall. I have two minor comments: # Why Acl#entries is a set? I'm wondering whether it should be ordered. # It seems to me that guava's Objects.equal() and Objects.hashCode() could be more efficient than Apache Common's EqualsBuilder() and HashCodeBuilder(). > FileSystem API for ACLs. > > > Key: HDFS-5594 > URL: https://issues.apache.org/jira/browse/HDFS-5594 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch > > > Add new methods to {{FileSystem}} for manipulating ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[ https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843410#comment-13843410 ] Colin Patrick McCabe commented on HDFS-4201: wouldn't it be easier to use a try+finally here to set bpNSInfo back to null if initBlockPool fails? That way, the code could call the normal methods like getNamespaceInfo. This may make it easier to merge this change with any other changes to the initialization path that are coming up. > NPE in BPServiceActor#sendHeartBeat > --- > > Key: HDFS-4201 > URL: https://issues.apache.org/jira/browse/HDFS-4201 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Eli Collins >Assignee: Jimmy Xiang >Priority: Critical > Fix For: 3.0.0 > > Attachments: trunk-4201.patch, trunk-4201_v2.patch > > > Saw the following NPE in a log. > Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not > {{bpRegistration}}) due to a configuration or local directory failure. > {code} > 2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of > 30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec; > heartBeatInterval=3000 > 2012-09-25 04:33:20,782 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService > for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id > DS-1031100678-11.164.162.251-5010-1341933415989) service to > svsrs00127/11.164.162.226:8020 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673) > at java.lang.Thread.run(Thread.java:722) > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5642) libhdfs Windows compatibility.
[ https://issues.apache.org/jira/browse/HDFS-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843404#comment-13843404 ] Chris Nauroth commented on HDFS-5642: - bq. I would prefer putting the Windows or Linux specific parts in separate files, rather than using #ifdefs, if possible. I agree with this in principle, and I would have preferred this pattern in libhadoop.so/hadoop.dll too. However, I seem to recall there was disagreement on this a while ago. Does anyone else have an opinion? bq. ...hash table... Agreed that the details of this are open for consideration, rather than locked down on choosing uthash. In fact, Stephen told me he encountered a compilation error in uthash on HP-UX, so perhaps it's not as cross-platform as we thought. > libhdfs Windows compatibility. > -- > > Key: HDFS-5642 > URL: https://issues.apache.org/jira/browse/HDFS-5642 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth > > Currently, the libhdfs codebase does not compile on Windows due to use of > several Linux-specific functions, lack of build script support, and use of > C99 constructs. The scope of this issue includes converting those function > calls to cross-platform equivalents (or use {{#ifdef}} if necessary), setting > up build support, and some code clean-ups to follow the C89 rules. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5644) Reduce lock contention in libhdfs global class reference hash table
[ https://issues.apache.org/jira/browse/HDFS-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5644: --- Summary: Reduce lock contention in libhdfs global class reference hash table (was: Reduce lock contention in libhdfs.) > Reduce lock contention in libhdfs global class reference hash table > --- > > Key: HDFS-5644 > URL: https://issues.apache.org/jira/browse/HDFS-5644 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth > > libhdfs has a hash table which contains class references. This hash table is > shared between threads and currently uses a single global lock. The scope of > this issue is to improve performance of libhdfs by removing this single > global lock. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5644) Reduce lock contention in libhdfs.
[ https://issues.apache.org/jira/browse/HDFS-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5644: --- Description: libhdfs has a hash table which contains class references. This hash table is shared between threads and currently uses a single global lock. The scope of this issue is to improve performance of libhdfs by removing this single global lock. (was: libhdfs uses locking internally for coordinating access to shared hash tables and the JNI environment. The scope of this issue is to improve performance of libhdfs by reducing lock contention.) > Reduce lock contention in libhdfs. > -- > > Key: HDFS-5644 > URL: https://issues.apache.org/jira/browse/HDFS-5644 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth > > libhdfs has a hash table which contains class references. This hash table is > shared between threads and currently uses a single global lock. The scope of > this issue is to improve performance of libhdfs by removing this single > global lock. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Assigned] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao reassigned HDFS-5632: --- Assignee: Jing Zhao > Add Snapshot feature to INodeDirectory > -- > > Key: HDFS-5632 > URL: https://issues.apache.org/jira/browse/HDFS-5632 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > > We will add snapshot feature to INodeDirectory and remove > INodeDirectoryWithSnapshot in this jira. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5644) Reduce lock contention in libhdfs.
[ https://issues.apache.org/jira/browse/HDFS-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843396#comment-13843396 ] Colin Patrick McCabe commented on HDFS-5644: libhdfs doesn't actually use locking to get the JNI environment. It uses thread-local data for that, since you need a different {{JNIEnv}} per thread. The shared hash table stores global class references. Since this is insert-only (we never remove elements), and we should not need to resize this if we choose the initial size properly, it would be relatively easy to create a concurrent version of this. > Reduce lock contention in libhdfs. > -- > > Key: HDFS-5644 > URL: https://issues.apache.org/jira/browse/HDFS-5644 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth > > libhdfs uses locking internally for coordinating access to shared hash tables > and the JNI environment. The scope of this issue is to improve performance > of libhdfs by reducing lock contention. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5642) libhdfs Windows compatibility.
[ https://issues.apache.org/jira/browse/HDFS-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843391#comment-13843391 ] Colin Patrick McCabe commented on HDFS-5642: I would prefer putting the Windows or Linux specific parts in separate files, rather than using #ifdefs, if possible. I agree that we should probably get rid of the C99 stuff like declaring variables in places other than the start of a block. There was a suggestion to use a library like uthash rather than the glibc hash table functions, since Windows doesn't have those. That could be a good idea. Alternately, we could just create our own hash table, since that is just not a very complex data structure. Having a statically sized hash table would probably be ok, since we should know roughly how many entries it should have based on looking at the code. > libhdfs Windows compatibility. > -- > > Key: HDFS-5642 > URL: https://issues.apache.org/jira/browse/HDFS-5642 > Project: Hadoop HDFS > Issue Type: Improvement > Components: libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Chris Nauroth > > Currently, the libhdfs codebase does not compile on Windows due to use of > several Linux-specific functions, lack of build script support, and use of > C99 constructs. The scope of this issue includes converting those function > calls to cross-platform equivalents (or use {{#ifdef}} if necessary), setting > up build support, and some code clean-ups to follow the C89 rules. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843292#comment-13843292 ] Yongjun Zhang commented on HDFS-4983: - Hi Daryn, Please see the comment Alejandro Abdelnur made on 05/Dec/13 01:01 above. Thanks. --Yongjun > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.4.0 > > Attachments: HDFS-4983.001.patch, HDFS-4983.002.patch, > HDFS-4983.003.patch, HDFS-4983.004.patch, HDFS-4983.005.patch, > HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5023) TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2
[ https://issues.apache.org/jira/browse/HDFS-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843287#comment-13843287 ] Mit Desai commented on HDFS-5023: - Jing, I have tried making the changes according your suggestion. That would work but this will cause problem with the two tests i) testAllowSnapshot ii) testNonSnapshotPathINodes. -These two tests does not create snapshots, so moving the "disallowSnapshot" and "deleteSnapshot" lines to "After" method, will throw an exception after their completion. -As a fix to that, we will have to create an unnecessary snapshot after these tests. -Or another fix would be to place the "disallowSnapshot" and "deleteSnapshot" lines in a try block and we don't do anything in the catch block ( This is just an additional option. But it does not seem to be good choice to me ). But, I think that none of these solutions can be good to go. What do you think? > TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2 > --- > > Key: HDFS-5023 > URL: https://issues.apache.org/jira/browse/HDFS-5023 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots, test >Affects Versions: 2.4.0 >Reporter: Ravi Prakash >Assignee: Mit Desai > Labels: test > Attachments: HDFS-5023.patch, HDFS-5023.patch, HDFS-5023.patch, > TEST-org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.xml, > org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes-output.txt > > > The assertion on line 91 is failing. I am using Fedora 19 + JDK7. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5318) Account for shared storage in NameNode replica counting algorithm
[ https://issues.apache.org/jira/browse/HDFS-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843254#comment-13843254 ] Eric Sirianni commented on HDFS-5318: - bq. Currently core Hadoop probably does not require this functionality. Should we consider pluggable interface to enable this? That seems reasonable. I'll look at what the right level of abstraction should be for this and take a first cut at a patch. > Account for shared storage in NameNode replica counting algorithm > - > > Key: HDFS-5318 > URL: https://issues.apache.org/jira/browse/HDFS-5318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Eric Sirianni > > There are several use cases for using shared-storage for datanode block > storage in an HDFS environment (storing cold blocks on a NAS device, Amazon > S3, etc.). > With shared-storage, there is a distinction between: > # a distinct physical copy of a block > # an access-path to that block via a datanode. > A single 'replication count' metric cannot accurately capture both aspects. > However, for most of the current uses of 'replication count' in the Namenode, > the "number of physical copies" aspect seems to be the appropriate semantic. > I propose altering the replication counting algorithm in the Namenode to > accurately infer distinct physical copies in a shared storage environment. > With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor > additional semantics to the {{StorageID}} - namely that multiple datanodes > attaching to the same physical shared storage pool should report the same > {{StorageID}} for that pool. A minor modification would be required in the > DataNode to enable the generation of {{StorageID}} s to be pluggable behind > the {{FsDatasetSpi}} interface. > With those semantics in place, the number of physical copies of a block in a > shared storage environment can be calculated as the number of _distinct_ > {{StorageID}} s associated with that block. > Consider the following combinations for two {{(DataNode ID, Storage ID)}} > pairs {{(DN_A, S_A) (DN_B, S_B)}} for a given block B: > * {{DN_A != DN_B && S_A != S_B}} - *different* access paths to *different* > physical replicas (i.e. the traditional HDFS case with local disks) > ** → Block B has {{ReplicationCount == 2}} > * {{DN_A != DN_B && S_A == S_B}} - *different* access paths to the *same* > physical replica (e.g. HDFS datanodes mounting the same NAS share) > ** → Block B has {{ReplicationCount == 1}} > For example, if block B has the following location tuples: > * {{DN_1, STORAGE_A}} > * {{DN_2, STORAGE_A}} > * {{DN_3, STORAGE_B}} > * {{DN_4, STORAGE_B}}, > the effect of this proposed change would be to calculate the replication > factor in the namenode as *2* instead of *4*. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843244#comment-13843244 ] Liang Xie commented on HDFS-1605: - Considering risk, i changes most of synchronized section with writeLock. only few places(like getFileLengh, some simple getter methods) use readLock in current impl, i think it's ok since both i and [~dhruba] observed the very same hotspot: getFileLengh, in similar scenario: HBase read. > Convert DFSInputStream synchronized sections to a ReadWrite lock > > > Key: HDFS-1605 > URL: https://issues.apache.org/jira/browse/HDFS-1605 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt, > HDFS-1605.txt > > > Hbase does concurrent preads from multiple threads to different blocks of the > same hdfs file. Each of these pread calls invoke > DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods > are "synchronized", thus causing all the concurrent threads to serialize. It > would help performance to convert this to a Read/Write lock -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-1605: Attachment: HDFS-1605.txt Attached is a patch against trunk. The getFileLength hotspot pattern like below: "IPC Server handler 88 on 12600" daemon prio=10 tid=0x7f82fc241580 nid=0x4ddc waiting for monitor entry [0x7f821eefb000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hdfs.DFSInputStream.getFileLength(DFSInputStream.java:242) - waiting to lock <0x0004404597a8> (a org.apache.hadoop.hdfs.DFSInputStream) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:982) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:73) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1393) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1829) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1673) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:341) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:485) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:506) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:226) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:146) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:354) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:385) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:344) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:304) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:584) - locked <0x0004285cb478> (a org.apache.hadoop.hbase.regionserver.StoreScanner) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:446) - locked <0x0004285cb478> (a org.apache.hadoop.hbase.regionserver.StoreScanner) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:164) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3658) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3590) - locked <0x0004285cb310> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3615) - locked <0x0004285cb310> (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:460) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1457) "IPC Server handler 192 on 12600" daemon prio=10 tid=0x7f82fc2dd910 nid=0x4e44 runnable [0x7f821d493000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:220) at org.apache.hadoop.hdfs.BlockReaderLocal.read(BlockReaderLocal.java:568) - locked <0x0004287fb350> (a org.apache.hadoop.hdfs.BlockReaderLocal) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:542) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:594) - locked <0x0004404597a8> (a org.apache.hadoop.hdfs.DFSInputStream) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:648) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689) - locked <0x0004404597a8> (a org.apache.hadoop.hdfs.DFSInputStream) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:614) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1384) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1829) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1673) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:341) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFile
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843240#comment-13843240 ] Liang Xie commented on HDFS-5637: - [~stack], seems no objection until now. > try to refeatchToken while local read InvalidToken occurred > --- > > Key: HDFS-5637 > URL: https://issues.apache.org/jira/browse/HDFS-5637 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, security >Affects Versions: 2.0.5-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5637.txt > > > we observed several warning logs like below from region server nodes: > 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to > connect to /10.2.201.110:11402 for block, add to deadNodes and continue. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with > block_token_identifier (expiryDate=1386060141977, keyId=-333530248, > userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, > blockId=-190217754078101701, access modes=[READ]) is expired. > at > org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) > at > org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) > at > org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) > at > org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) > at
[jira] [Commented] (HDFS-5639) rpc scheduler abstraction
[ https://issues.apache.org/jira/browse/HDFS-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843234#comment-13843234 ] Daryn Sharp commented on HDFS-5639: --- This patch seems excessively large to me - compounded by unnecessary changes such as using Thread.getCurrentThread when "this" is already a thread. Although conceptually different to some degree, it appears to overlap with HADOOP-9640. Hiding a scheduler behind a custom BlockingQueue implementation may be a bit less intrusive. Would you please work with Chris Li to see if there's enough similarity to combine these efforts (although still via separate jiras). > rpc scheduler abstraction > - > > Key: HDFS-5639 > URL: https://issues.apache.org/jira/browse/HDFS-5639 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma > Attachments: HDFS-5639-2.patch, HDFS-5639.patch > > > We have run into various issues in namenode and hbase w.r.t. rpc handling in > multi-tenant clusters. The examples are > https://issues.apache.org/jira/i#browse/HADOOP-9640 > https://issues.apache.org/jira/i#browse/HBASE-8836 > There are different ideas on how to prioritize rpc requests. It could be > based on user id, or whether it is read request or write request, or it could > use specific rule like datanode's RPC is more important than client RPC. > We want to enable people to implement and experiiment different rpc > schedulers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[ https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843218#comment-13843218 ] Daryn Sharp commented on HDFS-4983: --- I've questioned the necessity of this pattern before because it's required multiple tweaks since being added. Like RPC, why not rely on the UGI to delegate to the underlying OS for determining the validity of a username? > Numeric usernames do not work with WebHDFS FS > - > > Key: HDFS-4983 > URL: https://issues.apache.org/jira/browse/HDFS-4983 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Yongjun Zhang > Labels: patch > Fix For: 2.4.0 > > Attachments: HDFS-4983.001.patch, HDFS-4983.002.patch, > HDFS-4983.003.patch, HDFS-4983.004.patch, HDFS-4983.005.patch, > HDFS-4983.006.patch, HDFS-4983.006.patch > > > Per the file > hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java, > the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}. > Given this, using a username such as "123" seems to fail for some reason > (tried on insecure setup): > {code} > [123@host-1 ~]$ whoami > 123 > [123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls / > -ls: Invalid value: "123" does not belong to the domain > ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...] > {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)