Hi Steve
Thx for the input . Actually I have wrongly put the error stack
trace(complete error trace and other details are available in the Jira ,
apologies for the same. ) . The error is related to authenticatication
and it's happening only when the proxy user option is being used ,
otherwise things are working fine.
Also my intention wasn't to point out any mistakes or escalate the problem
to the community. I was just looking for help and direction to solve the
issue.
22/04/26 08:54:40 DEBUG Client: closing ipc connection to
<server>/<ip>:8020: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
at
org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
at
org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
at
org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
at
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at
org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
at
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364)
at scala.Option.map(Option.scala:230)
at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
Again Thx for the input and advice regarding documentation and
apologies for putting the wrong error stack earlier.
Regards
Pralabh Kumar
On Tue, May 3, 2022 at 7:39 PM Steve Loughran <ste...@cloudera.com> wrote:
>
> Prablah, did you follow the URL provided in the exception message? i put a
> lot of effort in to improving the diagnostics, where the wiki articles are
> part of the troubleshooing process
> https://issues.apache.org/jira/browse/HADOOP-7469
>
> it's really disappointing when people escalate the problem to open source
> developers before trying to fix the problem themselves, in this case, read
> the error message.
>
> now, if there is some k8s related issue which makes this more common, you
> are encouraged to update the wiki entry with a new cause. documentation is
> an important contribution to open source projects, and if you have
> discovered a new way to recreate the failure, it would be welcome. which
> reminds me, i have to add something to connection reset and docker which
> comes down to "turn off http keepalive in maven builds"
>
> -Steve
>
>
>
>
>
> On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi <gabor.g.somo...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Please be aware that ConnectionRefused exception is has nothing to do w/
>> authentication. See the description from Hadoop wiki:
>> "You get a ConnectionRefused
>> <https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused>
>> Exception
>> when there is a machine at the address specified, but there is no program
>> listening on the specific TCP port the client is using -and there is no
>> firewall in the way silently dropping TCP connection requests. If you do
>> not know what a TCP connection request is, please consult the
>> specification <http://www.ietf.org/rfc/rfc793.txt>."
>>
>> This means the namenode on host:port is not reachable in the TCP layer.
>> Maybe there are multiple issues but I'm pretty sure that something is wrong
>> in the K8S net config.
>>
>> BR,
>> G
>>
>>
>> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pralabhku...@gmail.com>
>> wrote:
>>
>>> Hi dev Team
>>>
>>> Spark-25355 added the functionality of the proxy user on K8s . However
>>> proxy user on K8s with Kerberized HDFS is not working . It is throwing
>>> exception and
>>>
>>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
>>> the server : org.apache.hadoop.security.AccessControlException: Client
>>> cannot authenticate via:[TOKEN, KERBEROS]
>>>
>>>
>>> Exception in thread "main" java.net.ConnectException: Call From
>>> <driverpod> to <namenode> failed on connection exception:
>>> java.net.ConnectException: Connection refused; For more details see: http:
>>> //wiki.apache.org/hadoop/ConnectionRefused
>>>
>>> at
>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>
>>> at
>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>> Source)
>>>
>>> at
>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>> Source)
>>>
>>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown
>>> Source)
>>>
>>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>>
>>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>>
>>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>>
>>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>>
>>> at
>>>
>>>
>>>
>>> On debugging deep , we found the proxy user doesn't have access to
>>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>>> the proxy user and this user doesn't have delegation token.
>>>
>>>
>>> Please help me with the same.
>>>
>>>
>>> Regards
>>>
>>> Pralabh Kumar
>>>
>>>
>>>
>>>
