subject:"\[jira\] \[Commented\] \(HDFS\-5637\) try to refeatchToken while local read InvalidToken occurred"

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2014-02-03 Thread Kihwal Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13889619#comment-13889619
 ] 

Kihwal Lee commented on HDFS-5637:
--

I pulled this into branch-0.23.

It fixes the issue for 2.0, but after the new domain socket based local reader, 
this fix won't work for the old legacy short-circuit local reader. This is 
because {{DFSInputStream#getBlockReader()}} catches {{IOException}} and 
disables it when calling {{BlockReaderFactory.getLegacyBlockReaderLocal()}}.  



 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Fix For: 0.23.11, 2.3.0

 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2014-02-03 Thread Kihwal Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13889636#comment-13889636
 ] 

Kihwal Lee commented on HDFS-5637:
--

Filed HDFS-5870.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Fix For: 0.23.11, 2.3.0

 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847398#comment-13847398
 ] 

Hudson commented on HDFS-5637:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #420 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/420/])
HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. (Liang 
Xie via junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550335)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java


 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Fix For: 2.4.0

 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847544#comment-13847544
 ] 

Hudson commented on HDFS-5637:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1637 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1637/])
HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. (Liang 
Xie via junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550335)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java


 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Fix For: 2.4.0

 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-12 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846373#comment-13846373
 ] 

Hudson commented on HDFS-5637:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #1610 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1610/])
HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. (Liang 
Xie via junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550335)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java


 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Fix For: 2.4.0

 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846049#comment-13846049
 ] 

Hudson commented on HDFS-5637:
--

FAILURE: Integrated in Hadoop-trunk-Commit #4870 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/4870/])
HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. (Liang 
Xie via junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1550335)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java


 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Fix For: 2.4.0

 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844091#comment-13844091
 ] 

Hadoop QA commented on HDFS-5637:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12617994/HDFS-5637-v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/5686//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5686//console

This message is automatically generated.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844565#comment-13844565
 ] 

Colin Patrick McCabe commented on HDFS-5637:


+1.  thanks, Liang

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844610#comment-13844610
 ] 

stack commented on HDFS-5637:
-

If good w/ you [~djp], I will go ahead and commit.  Thanks.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845014#comment-13845014
 ] 

Junping Du commented on HDFS-5637:
--

The purpose of refactoring is trying to keep consistent on exception handling 
for different cases. If we think abstract a method for twice usage is too 
overkill, I think we should at least add some comments, so later developer and 
reviewer should pay attention on this when adding additional exceptions. 
Thoughts?

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845019#comment-13845019
 ] 

Colin Patrick McCabe commented on HDFS-5637:


I agree there could be some refactoring, but I think it would be best to do 
that in a separate JIRA.  Some people might just want the fix for this.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845027#comment-13845027
 ] 

Liang Xie commented on HDFS-5637:
-

I just filed HDFS-5652 about refactoring or uniforming the invalid token 
handling in DFSInputStream.
I didn't assign it to me right now due to there're other blocking stuff need be 
done asap,  will work on it later if nobody pick it up.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845033#comment-13845033
 ] 

Colin Patrick McCabe commented on HDFS-5637:


ok, thanks.  +1 on this change, then.  thanks, all

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-10 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845055#comment-13845055
 ] 

Junping Du commented on HDFS-5637:
--

+1. I am OK with doing work in separated JIRA. Thanks Liang for the patch!

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-09 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843240#comment-13843240
 ] 

Liang Xie commented on HDFS-5637:
-

[~stack], seems no objection until now.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-09 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843550#comment-13843550
 ] 

Colin Patrick McCabe commented on HDFS-5637:


is there any reason why you didn't change it in fetchBlockByteRange as well?

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-09 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843907#comment-13843907
 ] 

Junping Du commented on HDFS-5637:
--

Thanks [~xieliang007] for the patch! Agree with Colin that handling exception 
in getBlockReader() in fetchBlockByteRange should be the same. In addition, I 
would suggest to refactor the exception handling code here to be a method and 
get shared in places calling getBlockReader(). How it sounds?

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-09 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843999#comment-13843999
 ] 

Liang Xie commented on HDFS-5637:
-

Good catch, V2 should address comments from Colin and Junping,  seems i am too 
lazy to do a grep before make a patch...
For refactoring, it doesn't repeat more than twice and the code section is 
pretty short, to me it still ok for leaving it unchanged:)

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637-v2.txt, HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-06 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841110#comment-13841110
 ] 

Hadoop QA commented on HDFS-5637:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12617342/HDFS-5637.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/5656//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5656//console

This message is automatically generated.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-06 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841883#comment-13841883
 ] 

stack commented on HDFS-5637:
-

+1 on the patch.  Will commit in next few days unless objection.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-05 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841017#comment-13841017
 ] 

Liang Xie commented on HDFS-5637:
-

Our hadoop version is  based on 2.0.0 modified.
The log shows the addToDeadNodes() will be triggered if a ssr failed due to 
token expired, we should try to refetchToken first, just like 
InvalidBlockTokenException occurred.
Attached patch is a quick/simple fix, just keep the similar handle like 
InvalidBlockTokenException.
Another possible fix is that we should uniform into one same exception for both 
ssr and normal non-ssr scenario.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-05 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841020#comment-13841020
 ] 

Liang Xie commented on HDFS-5637:
-

Specially, [~apurtell], would you mind taking a look at it as well? thanks, i 
know you are definitely a HBase security expert, probably you have seen it 
before in any region server nodes? 

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

2013-12-05 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13841071#comment-13841071
 ] 

Andrew Purtell commented on HDFS-5637:
--

+1

Looks like an update needed for Hadoop 2.x that was missed.

 try to refeatchToken while local read InvalidToken occurred
 ---

 Key: HDFS-5637
 URL: https://issues.apache.org/jira/browse/HDFS-5637
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, security
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HDFS-5637.txt


 we observed several warning logs like below from region server nodes:
 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to 
 connect to /10.2.201.110:11402 for block, add to deadNodes and continue. 
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with 
 block_token_identifier (expiryDate=1386060141977, keyId=-333530248, 
 userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, 
 blockId=-190217754078101701, access modes=[READ]) is expired.
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280)
 at 
 org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082)
 at 
 org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112)
 at 
 org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
 Method)
 at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
 at

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred

23 matches

Site Navigation

Mail list logo

Footer information