Karthik Palanisamy created HADOOP-19732:
-------------------------------------------
Summary: NameNode Crash Due to Delegation Renewer Runtime Exit
(NoMatchingRule)
Key: HADOOP-19732
URL: https://issues.apache.org/jira/browse/HADOOP-19732
Project: Hadoop Common
Issue Type: Improvement
Reporter: Karthik Palanisamy
The delegation token renewer enters runtime exit when a _NoMatchingRule_ error,
which caused the entire namenode to crash. I think returning an error to the
client should be fine but bringing down the namenode is not acceptable to
anyone.
After the AD change, the new realm was updated, but some jobs are still using
the old realm as users are updating them gradually. This migration process will
take time, and during this period, other jobs are still catching up with the
new realm configuration. However, the namenode down disrupts all of them.
{code:java}
ERROR
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
ExpiredTokenRemover thread received unexpected exception
java.lang.IllegalArgumentException: Illegal principal name hive/[email protected]
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to hive/[email protected] at
org.apache.hadoop.security.User.<init>(User.java:51)
at org.apache.hadoop.security.User.<init>(User.java:43)
at
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1417)
at
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1401)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier.getUser(AbstractDelegationTokenIdentifier.java:80)
at
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier.getUser(DelegationTokenIdentifier.java:81)
at
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier.toString(DelegationTokenIdentifier.java:91)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:137)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.formatTokenId(AbstractDelegationTokenSecretManager.java:58)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.logExpireTokens(AbstractDelegationTokenSecretManager.java:642)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.removeExpiredToken(AbstractDelegationTokenSecretManager.java:635)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.access$400(AbstractDelegationTokenSecretManager.java:51)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:694)
at java.lang.Thread.run(Thread.java:750)
Caused by:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to hive/[email protected]
at
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:429)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 14 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
