[
https://issues.apache.org/jira/browse/HDFS-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Xiaoqiao updated HDFS-10728:
-------------------------------
Affects Version/s: 2.7.1
Description:
In security authentication hadoop cluster with Kerberos, when I changed SBN's
ip address, hostname not changed and the routing tables are also refresh, DN
can't communicate with NN since auth failed.
Trace info of DN as following:
{code:xml}
2016-08-04 22:45:25,719 WARN org.apache.hadoop.ipc.Client: Address change
detected. Old: {hostname}/{ip1}:8020 New: {hostname}/{ip2}:8020
2016-08-04 22:45:25,719 INFO org.apache.hadoop.ipc.Client: Retrying connect to
server: {hostname}/{ip2}:8020. Already tried 0 time(s); maxRetries=45
......
2016-08-04 22:45:38,010 WARN org.apache.hadoop.ipc.Client: Couldn't setup
connection for hdfs/{dn hostname}@REALMS.COM to {hostname}/{ip1}:8020
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
initiate failed
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
at org.apache.hadoop.ipc.Client.call(Client.java:1446)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
at java.lang.Thread.run(Thread.java:745)
2016-08-04 22:45:38,014 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
IOException in offerService
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for hdfs/{dn hostname}@REALMS.COM to {hostname}/{ip1}:8020;
Host Details : local host is: "{dn hostname}/{dn ip}"; destination host is:
"{hostname}":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1474)
at org.apache.hadoop.ipc.Client.call(Client.java:1407)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't setup connection for hdfs/{dn
hostname}@REALMS.COM to {hostname}/{ip1}:8020
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
at org.apache.hadoop.ipc.Client.call(Client.java:1446)
... 8 more
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
initiate failed
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
... 11 more
{code}
Component/s: ipc
Summary: DN can't communicate with NN when IP address of NN
changed since Auth failed (was: DataNode cann't connect to NameNode)
> DN can't communicate with NN when IP address of NN changed since Auth failed
> ----------------------------------------------------------------------------
>
> Key: HDFS-10728
> URL: https://issues.apache.org/jira/browse/HDFS-10728
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ipc
> Affects Versions: 2.7.1
> Reporter: He Xiaoqiao
>
> In security authentication hadoop cluster with Kerberos, when I changed SBN's
> ip address, hostname not changed and the routing tables are also refresh, DN
> can't communicate with NN since auth failed.
> Trace info of DN as following:
> {code:xml}
> 2016-08-04 22:45:25,719 WARN org.apache.hadoop.ipc.Client: Address change
> detected. Old: {hostname}/{ip1}:8020 New: {hostname}/{ip2}:8020
> 2016-08-04 22:45:25,719 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: {hostname}/{ip2}:8020. Already tried 0 time(s); maxRetries=45
> ......
> 2016-08-04 22:45:38,010 WARN org.apache.hadoop.ipc.Client: Couldn't setup
> connection for hdfs/{dn hostname}@REALMS.COM to {hostname}/{ip1}:8020
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
> at org.apache.hadoop.ipc.Client.call(Client.java:1446)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
> at java.lang.Thread.run(Thread.java:745)
> 2016-08-04 22:45:38,014 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in offerService
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for hdfs/{dn hostname}@REALMS.COM to {hostname}/{ip1}:8020;
> Host Details : local host is: "{dn hostname}/{dn ip}"; destination host is:
> "{hostname}":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
> at org.apache.hadoop.ipc.Client.call(Client.java:1474)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Couldn't setup connection for hdfs/{dn
> hostname}@REALMS.COM to {hostname}/{ip1}:8020
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
> at org.apache.hadoop.ipc.Client.call(Client.java:1446)
> ... 8 more
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
> ... 11 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
