Hello Dmitry,
For a successful SASL authentication, both the client and server need to
negotiate and come to agreement on a quality of protection. The Hadoop
configuration property hadoop.rpc.protection supports a comma-separate list of
values that map to the SASL QoP values.
<property>
<name>hadoop.rpc.protection</name>
<value>authentication</value>
<description>A comma-separated list of protection values for secured sasl
connections. Possible values are authentication, integrity and privacy.
authentication means authentication only and no integrity or privacy;
integrity implies authentication and integrity are enabled; and privacy
implies all of authentication, integrity and privacy are enabled.
hadoop.security.saslproperties.resolver.class can be used to override
the hadoop.rpc.protection for a connection at the server side.
</description>
</property>
This means that for this application to work, both the client end and the
server end must have at least one value in common in that list. Judging from
the stack trace, it looks like the client is trying to communicate with the
HDFS NameNode, so I recommend that you review the setting for this
configuration property at the NameNode and the application. If you change the
configuration property for the NameNode, then it will require a restart for the
change to take effect.
If this still isn't working, then it's possible that Java's SASL debugging
might provide more clues. Details on how to enable that logging are here:
https://docs.oracle.com/javase/8/docs/technotes/guides/security/sasl/sasl-refguide.html#DEBUG
--Chris Nauroth
From: Dmitry Goldenberg
<dgoldenberg...@gmail.com<mailto:dgoldenberg...@gmail.com>>
Date: Wednesday, June 8, 2016 at 4:30 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: SaslException: No common protection layer between client and server ?
I'm trying to communicate programmatically to a Hadoop cluster which is
kerberized (CDH 5.3/HDFS 2.5.0).
I have a valid Kerberos token on the client side. But I'm getting an error as
below, "No common protection layer between client and server".
What does this error mean and are there any ways to fix or work around it?
Is this something related to
HDFS-5688<https://issues.apache.org/jira/browse/HDFS-5688>? The ticket seems to
imply that the property "hadoop.rpc.protection" must be set, presumably to
"authentication<https://datameer.zendesk.com/hc/en-us/articles/204262060-Errors-when-Executing-Job-Caused-by-javax-security-sasl-SaslException-No-common-protection-layer-between-client-and-server>"
(also per e.g.
this<https://datameer.zendesk.com/hc/en-us/articles/204262060-Errors-when-Executing-Job-Caused-by-javax-security-sasl-SaslException-No-common-protection-layer-between-client-and-server>).
Would this need to be set on all servers in the cluster and then the cluster
bounced? I don't have easy access to the cluster so I need to understand
whether 'hadoop.rpc.protection' is the actual cause. It seems that
'authentication' should be the value used by default, at least according to the
core-default.xml
documentation<https://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/core-default.xml>.
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for
principal1/server1.acme....@xxx.acme.net<mailto:bdgtr001a01i1d.nam.nsroot....@namuxdev.dyn.nsroot.net>
to
server2.acme.net/10.XX.XXX.XXX:8020<http://bdgtmaster02i1d.nam.nsroot.net/10.59.123.140:8020>;
Host Details : local host is:
“some-host.acme.net/168.XX.XXX.XX<http://sd-e13b-c041.nam.nsroot.net/168.72.192.72>”;
destination host is:
“server2.acm<http://bdgtmaster02i1d.nam.nsroot.net/>e.net<http://e.net>”:8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)
... 11 more
Caused by: java.io.IOException: Couldn't setup connection for
principal1/server1.acme....@xxx.acme.net<mailto:bdgtr001a01i1d.nam.nsroot....@namuxdev.dyn.nsroot.net>
to
server2.acme.net/10.XX.XXX.XXX:8020<http://bdgtmaster02i1d.nam.nsroot.net/10.59.123.140:8020>;
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
... 31 more
Caused by: javax.security.sasl.SaslException: No common protection layer
between client and server
at
com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:251)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:186)
at
org.apache.hadoop.security.SaslRpcClient.saslEvaluateToken(SaslRpcClient.java:483)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:427)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
... 34 more
