Hi,
I have a Kerberos setup with Hadoop (single node cluster) in an Ubuntu
environment (VirtualBox setup).
We are using a variant of a Yarn application and the Client.java in this
variant opens a socket for communicating to the ApplicationMaster and
receiving messages.
Without Kerberos, this works fine. I am currently investigating whether the
entire structure will work with Kerberos too and what code changes would be
necessary. With Kerberos, a problem occurs at the socket connection part
and simply fails with errors outlined in the attached file
(kerbFailure.txt), a snippet of the errors is:
16/03/18 17:18:28 WARN ipc.Client: Exception encountered while connecting
to the server : org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[KERBEROS]
Now I have enabled Kerberos authentication on the Hadoop cluster by
following the instructions at:
http://www.cloudera.com/documentation/archive/cdh/4-x/4-3-0/CDH4-Security-Guide/cdh4sg_topic_3.html
Since the stacktrace has references to SASL connection methods, should I
explicitly enable SASL authentication, following the instructions at:
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SecureMode.html
?
My impression was that SASL DataTransfer is optional (only needed if I want
to start Nodes as non-root) and I currently start up the Secure Data Node
as root and set JSVC_HOME, using the scripts in the sbin folder.
I can also verify that both client and server processes return the correct
Kerberos principal when I do:
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
LOG.info("UGI: " + ugi + ", hasKerb: " + ugi.hasKerberosCredentials());
// outputs: UGI: ram@RAM-VIRTUALBOX (auth:KERBEROS), hasKerb: true
I have hdfs and yarn as separate users. Both have their relevant Kerberos
principals and authenticated through keytabs. My username is added as a
principal too and authenticated with a password. So system startup and Yarn
job submission is fine, but I encounter the error at socket connection as
described before.
Hope this overview helps. Please let me know if you might need more
information.
Thanking You,
K.N.Ramachandran
16/03/18 17:18:28 INFO util.Utilities: Connecting to ApplicationMaster at
ram-VirtualBox/127.0.1.1:34718
16/03/18 17:18:28 INFO ipc.Client: Connecting to ram-VirtualBox/127.0.1.1:34718
16/03/18 17:18:28 INFO security.SaslRpcClient: Checking SaslClient,
isClientNull: true
16/03/18 17:18:28 WARN ipc.Client: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[KERBEROS]
Caused by: java.io.IOException: Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[KERBEROS]; Host Details : local host is: "ram-VirtualBox/127.0.1.1";
destination host is: "ram-VirtualBox":34718;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
at org.apache.hadoop.ipc.Client.call(Client.java:1432)
at org.apache.hadoop.ipc.Client.call(Client.java:1359)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:242)
... 10 more
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:685)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:648)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:736)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1494)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
... 12 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[KERBEROS]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:404)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:723)