Hari Sekhon created AMBARI-8785:
-----------------------------------
Summary: Resource Manager HA Kerberos principal not handled or
even notified
Key: AMBARI-8785
URL: https://issues.apache.org/jira/browse/AMBARI-8785
Project: Ambari
Issue Type: Bug
Affects Versions: 1.7.0
Environment: HDP 2.1
Reporter: Hari Sekhon
When enabling Yarn Resource Manager HA in a Kerberos secured cluster, Ambari
fails to tell the user about the required Kerberos principal + keytab for the
new Resource Manager instance.
As as result the new Resource Manager fails to start with the following logs:
{code}2014-12-18 11:39:06,379 FATAL resourcemanager.ResourceManager
(ResourceManager.java:main(1043)) - Error starting ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:910)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1041)
Caused by: java.io.IOException: Login failure for rm/<host>@REALM from keytab
/etc/security/keytabs/rm.service.keytab
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:920)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:929)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:908)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password
from user
at
com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:911)
... 5 more
2014-12-18 11:39:06,383 INFO resourcemanager.ResourceManager
(StringUtils.java:run(640)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at <host>/x.x.x.x
************************************************************/
{code}
The fix is quite simple for experienced cluster administrators, create the
principal, export the keytab, mimic the original RM permissions of yarn:hadoop
620 (not really correct octal but that's a separate issue I've already raised
to fix keytab perms AMBARI-8138) and then it will start up.
This is sort of related to AMBARI-8610 where Ambari should export a CSV for new
hosts/services to allow existing automation scripts to generate the principals
and and distributed the keytabs such as the basic kerberos_setup.sh shipped
with Ambari or the more real world FreeIPA one I publish on my github
(https://github.com/harisekhon/toolbox).
Except in this instance it's not a new service, it's just HA enablement where
it doesn't notify users that RM HA will require a new Keberos principal +
keytab to allow them to do that, never mind provide a subset CSV for passing to
a kerberos setup script. This issue will also affect NN HA if Ambari ever
allows setting up NN HA without disabling Kerberos first.
Regards,
Hari Sekhon
http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
