[jira] [Commented] (YARN-241) Node Manager fails to launch containers after NM restart in secure mode
[
https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704088#comment-13704088
]
Omkar Vinit Joshi commented on YARN-241:
[~devaraj.k] closing it as not reproducible. lets open it if this issue shows
up again.
> Node Manager fails to launch containers after NM restart in secure mode
> ---
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Assignee: Omkar Vinit Joshi
>Priority: Critical
>
> After restarting the Node Manager it fails to launch containers with the
> below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8048: readAndProcess threw exception
> java.lang.IllegalArgumentException: Invalid key to HMAC computation from
> client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at
> org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports
> this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch containers after NM restart in secure mode
[
https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704081#comment-13704081
]
Devaraj K commented on YARN-241:
Omkar, For me also it was not happening always. When I ran the cluster for long
time and NM gets restarted multiple times I had faced it, when the issue comes
it fails to launch all containers on that node due to error mentioned in the
description.
> Node Manager fails to launch containers after NM restart in secure mode
> ---
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Assignee: Omkar Vinit Joshi
>Priority: Critical
>
> After restarting the Node Manager it fails to launch containers with the
> below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8048: readAndProcess threw exception
> java.lang.IllegalArgumentException: Invalid key to HMAC computation from
> client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at
> org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports
> this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch containers after NM restart in secure mode
[
https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13703977#comment-13703977
]
Omkar Vinit Joshi commented on YARN-241:
I tried restarting NM several times in secured mode.. but containers
successfully start /finish on it. However now this authentication happens
irrespective of security and instead of container tokens, nmtokens are getting
used.
> Node Manager fails to launch containers after NM restart in secure mode
> ---
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Assignee: Omkar Vinit Joshi
>Priority: Critical
>
> After restarting the Node Manager it fails to launch containers with the
> below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8048: readAndProcess threw exception
> java.lang.IllegalArgumentException: Invalid key to HMAC computation from
> client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at
> org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports
> this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch containers after NM restart in secure mode
[
https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13703961#comment-13703961
]
Omkar Vinit Joshi commented on YARN-241:
[~devaraj.k] taking over... any idea how to reproduce this?
> Node Manager fails to launch containers after NM restart in secure mode
> ---
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Assignee: Omkar Vinit Joshi
>Priority: Critical
>
> After restarting the Node Manager it fails to launch containers with the
> below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8048: readAndProcess threw exception
> java.lang.IllegalArgumentException: Invalid key to HMAC computation from
> client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at
> org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports
> this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch containers after NM restart in secure mode
[
https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503867#comment-13503867
]
Devaraj K commented on YARN-241:
As per my observation when I debug, It is having the secret key and
mac.init(key) is failing. It fails for all subsequent invocations. If we try
with new mac instance with same secret key it succeeds.
> Node Manager fails to launch containers after NM restart in secure mode
> ---
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Priority: Blocker
>
> After restarting the Node Manager it fails to launch containers with the
> below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8048: readAndProcess threw exception
> java.lang.IllegalArgumentException: Invalid key to HMAC computation from
> client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at
> org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports
> this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch containers after NM restart in secure mode
[
https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503836#comment-13503836
]
Daryn Sharp commented on YARN-241:
--
Is this maybe caused by a race condition where the NM is receiving a container
token before the RM registration completes and it receives the secret keys for
the container tokens?
> Node Manager fails to launch containers after NM restart in secure mode
> ---
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Priority: Blocker
>
> After restarting the Node Manager it fails to launch containers with the
> below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 8048: readAndProcess threw exception
> java.lang.IllegalArgumentException: Invalid key to HMAC computation from
> client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at
> org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at
> org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at
> org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at
> org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports
> this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at
> org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
