[
https://issues.apache.org/jira/browse/YARN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744121#comment-13744121
]
Arpit Gupta commented on YARN-1082:
-----------------------------------
Here are the logs
{code}
2013-08-17 17:32:08,272 INFO resourcemanager.ResourceManager
(SignalLogger.java:register(91)) - registered UNIX signal handlers for [TERM,
HUP, INT]
2013-08-17 17:32:08,544 DEBUG service.AbstractService
(AbstractService.java:enterState(452)) - Service: ResourceManager entered state
INITED
2013-08-17 17:32:08,683 DEBUG service.CompositeService
(CompositeService.java:addService(69)) - Adding service Dispatcher
2013-08-17 17:32:08,685 INFO security.AMRMTokenSecretManager
(AMRMTokenSecretManager.java:rollMasterKey(105)) - Rolling master-key for
amrm-tokens
2013-08-17 17:32:08,690 DEBUG service.CompositeService
(CompositeService.java:addService(69)) - Adding service
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
2013-08-17 17:32:08,691 DEBUG service.CompositeService
(CompositeService.java:addService(69)) - Adding service AMLivelinessMonitor
2013-08-17 17:32:08,691 DEBUG service.CompositeService
(CompositeService.java:addService(69)) - Adding service AMLivelinessMonitor
2013-08-17 17:32:08,694 DEBUG service.CompositeService
(CompositeService.java:addService(69)) - Adding service
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer
2013-08-17 17:32:08,699 INFO security.RMContainerTokenSecretManager
(RMContainerTokenSecretManager.java:<init>(75)) -
ContainerTokenKeyRollingInterval: 86400000ms and
ContainerTokenKeyActivationDelay: 900000ms
2013-08-17 17:32:08,704 INFO security.NMTokenSecretManagerInRM
(NMTokenSecretManagerInRM.java:<init>(77)) - NMTokenKeyRollingInterval:
86400000ms and NMTokenKeyActivationDelay: 900000ms
2013-08-17 17:32:08,738 DEBUG service.AbstractService
(AbstractService.java:enterState(452)) - Service: Dispatcher entered state
INITED
2013-08-17 17:32:08,847 INFO event.AsyncDispatcher
(AsyncDispatcher.java:register(157)) - Registering class
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreEventType
for class
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler
2013-08-17 17:32:08,848 DEBUG service.AbstractService
(AbstractService.java:start(197)) - Service Dispatcher is started
2013-08-17 17:32:09,084 DEBUG security.Groups
(Groups.java:getUserToGroupsMappingService(180)) - Creating new Groups object
2013-08-17 17:32:09,088 DEBUG util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(46)) - Trying to load the custom-built
native-hadoop library...
2013-08-17 17:32:09,089 DEBUG util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(50)) - Loaded the native-hadoop library
2013-08-17 17:32:09,089 DEBUG security.JniBasedUnixGroupsMapping
(JniBasedUnixGroupsMapping.java:<clinit>(50)) - Using JniBasedUnixGroupsMapping
for Group resolution
2013-08-17 17:32:09,090 DEBUG security.JniBasedUnixGroupsMappingWithFallback
(JniBasedUnixGroupsMappingWithFallback.java:<init>(44)) - Group mapping
impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
2013-08-17 17:32:09,090 DEBUG security.Groups (Groups.java:<init>(66)) - Group
mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback;
cacheTimeout=300000
2013-08-17 17:32:09,097 DEBUG security.UserGroupInformation
(UserGroupInformation.java:login(176)) - hadoop login
2013-08-17 17:32:09,097 DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(125)) - hadoop login commit
2013-08-17 17:32:09,098 DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(139)) - using kerberos user:null
2013-08-17 17:32:09,099 DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(155)) - using local user:UnixPrincipal: yarn
2013-08-17 17:32:09,101 DEBUG security.UserGroupInformation
(UserGroupInformation.java:getLoginUser(696)) - UGI loginUser:yarn
(auth:KERBEROS)
2013-08-17 17:32:09,216 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(326)) - dfs.client.use.legacy.blockreader.local = false
2013-08-17 17:32:09,217 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(329)) - dfs.client.read.shortcircuit = true
2013-08-17 17:32:09,217 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(332)) - dfs.client.domain.socket.data.traffic = false
2013-08-17 17:32:09,217 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(335)) - dfs.domain.socket.path =
/var/lib/hadoop-hdfs/dn_socket
2013-08-17 17:32:09,234 DEBUG hdfs.HAUtil
(HAUtil.java:cloneDelegationTokenForLogicalUri(276)) - No HA service delegation
token found for logical URI hdfs://host/apps/yarn/recovery
2013-08-17 17:32:09,235 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(326)) - dfs.client.use.legacy.blockreader.local = false
2013-08-17 17:32:09,235 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(329)) - dfs.client.read.shortcircuit = true
2013-08-17 17:32:09,235 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(332)) - dfs.client.domain.socket.data.traffic = false
2013-08-17 17:32:09,235 DEBUG hdfs.BlockReaderLocal
(DFSClient.java:<init>(335)) - dfs.domain.socket.path =
/var/lib/hadoop-hdfs/dn_socket
2013-08-17 17:32:09,244 DEBUG retry.RetryUtils
(RetryUtils.java:getDefaultRetryPolicy(74)) - multipleLinearRandomRetry = null
2013-08-17 17:32:09,262 DEBUG ipc.Server
(Server.java:registerProtocolEngine(220)) - rpcKind=RPC_PROTOCOL_BUFFER,
rpcRequestWrapperClass=class
org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper,
rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@700a4488
2013-08-17 17:32:09,483 DEBUG hdfs.BlockReaderLocal
(DomainSocketFactory.java:<init>(72)) - The short-circuit local reads feature
is enabled.
2013-08-17 17:32:09,495 DEBUG hdfs.DFSClient
(DFSClient.java:primitiveMkdir(2347)) -
/apps/yarn/recovery/FSRMStateRoot/RMDTSecretManagerRoot: masked=rwxr-xr-x
2013-08-17 17:32:09,518 DEBUG ipc.Client (Client.java:<init>(371)) - The ping
interval is 60000 ms.
2013-08-17 17:32:09,519 DEBUG ipc.Client (Client.java:setupIOstreams(636)) -
Connecting to host/ip:8020
2013-08-17 17:32:09,534 DEBUG security.UserGroupInformation
(UserGroupInformation.java:logPrivilegedAction(1499)) - PrivilegedAction
as:yarn (auth:KERBEROS)
from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:654)
2013-08-17 17:32:09,608 DEBUG security.SaslRpcClient
(SaslRpcClient.java:sendSaslMessage(438)) - Sending sasl message state:
NEGOTIATE
2013-08-17 17:32:09,614 DEBUG security.SaslRpcClient
(SaslRpcClient.java:saslConnect(370)) - Received SASL message state: NEGOTIATE
auths {
method: "TOKEN"
mechanism: "DIGEST-MD5"
protocol: ""
serverId: "default"
challenge:
"realm=\"default\",nonce=\"3tkm4i376oI+x2Y3d3bkbwXOLI1jp0O/0GRzA2UL\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
}
auths {
method: "KERBEROS"
mechanism: "GSSAPI"
protocol: "nn"
serverId: "host"
}
2013-08-17 17:32:09,615 DEBUG security.SaslRpcClient
(SaslRpcClient.java:getServerToken(259)) - Get token info proto:interface
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
info:@org.apache.hadoop.security.token.TokenInfo(value=class
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)
2013-08-17 17:32:09,616 DEBUG security.SaslRpcClient
(SaslRpcClient.java:getServerPrincipal(287)) - Get kerberos info
proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
serverPrincipal=dfs.namenode.kerberos.principal)
2013-08-17 17:32:09,617 DEBUG security.SaslRpcClient
(SaslRpcClient.java:createSaslClient(231)) - RPC Server's Kerberos principal
name for protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is
nn/h...@example.com
2013-08-17 17:32:09,617 DEBUG security.SaslRpcClient
(SaslRpcClient.java:createSaslClient(242)) - Creating SASL GSSAPI(KERBEROS)
client to authenticate to service at host
2013-08-17 17:32:09,624 DEBUG security.SaslRpcClient
(SaslRpcClient.java:selectSaslClient(172)) - Use KERBEROS authentication for
protocol ClientNamenodeProtocolPB
2013-08-17 17:32:09,626 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1480)) - PriviledgedActionException as:yarn
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
2013-08-17 17:32:09,626 DEBUG security.UserGroupInformation
(UserGroupInformation.java:logPrivilegedAction(1499)) - PrivilegedAction
as:yarn (auth:KERBEROS)
from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:583)
2013-08-17 17:32:09,627 WARN ipc.Client (Client.java:run(615)) - Exception
encountered while connecting to the server : javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
2013-08-17 17:32:09,627 ERROR security.UserGroupInformation
(UserGroupInformation.java:doAs(1480)) - PriviledgedActionException as:yarn
(auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
2013-08-17 17:32:09,628 DEBUG ipc.Client (Client.java:close(1099)) - closing
ipc connection to host/ip:8020: javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:620)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:583)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:667)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy9.mkdirs(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at $Proxy10.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2350)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2321)
at
org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:828)
at
org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:824)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:824)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:817)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1929)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.initInternal(FileSystemRMStateStore.java:93)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.init(RMStateStore.java:191)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:190)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:826)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:394)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:494)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:314)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:659)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:655)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:654)
... 27 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
... 36 more
2013-08-17 17:32:09,633 DEBUG ipc.Client (Client.java:close(1107)) - IPC Client
(533000440) connection to host/ip:8020 from yarn: closed
2013-08-17 17:32:09,637 DEBUG retry.RetryInvocationHandler
(RetryInvocationHandler.java:invoke(139)) - Exception while invoking mkdirs of
class ClientNamenodeProtocolTranslatorPB. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.S
{code}
> Secure RM with recovery enabled and rm state store on hdfs fails with gss
> exception
> -----------------------------------------------------------------------------------
>
> Key: YARN-1082
> URL: https://issues.apache.org/jira/browse/YARN-1082
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.1.0-beta
> Reporter: Arpit Gupta
> Assignee: Jian He
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
