Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Gabor Somogyi 
If you take a look at the jiras then ..33 is the newly opened same issue
which is discussed on ..55.
The quality of the logs are the exact same which is useless for real
analysis...

Not going too far spark-submit has cluster mode, the logs are partial and
contains client mode (so they don't belong together).

> By having a good JIRA where the problem is discussed and the same with
the PR description helps when others try to change the code in the future.


Not sure what you mean here. The guys who made the changes understands the
feature/change which is not necessarily true for all others.
This is double true for security features which is super hairy area...

G


On Tue, May 3, 2022 at 9:28 PM Bjørn Jørgensen 
wrote:

> What is the JIRA ticket for this problem?
>
> SPARK-25355  was
> marked as resolved for 2 years ago. But now there are a lot of new comments
> on whether things work and not.
>
> SPARK-39033 Support --proxy-user for Spark on K8s not working
>  is
> this yours problem?
>
> By having a good JIRA where the problem is discussed and the same with the
> PR description helps when others try to change the code in the future.
>
>
> tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar :
>
>> Hi Steve
>>
>> Thx for the input  . Actually I have wrongly put the error stack
>> trace(complete error trace and other details are available in the Jira ,
>> apologies for the same. )  .  The error is related to authenticatication
>> and it's happening only when the proxy user option is being used ,
>> otherwise things are working fine.
>>
>> Also my intention wasn't to point out any mistakes or
>> escalate the problem to the community.   I was just looking for help and
>> direction to solve the issue.
>>
>> 22/04/26 08:54:40 DEBUG Client: closing ipc connection to 
>> /:8020: org.apache.hadoop.security.AccessControlException: 
>> Client cannot authenticate via:[TOKEN, KERBEROS]
>> java.io.IOException: org.apache.hadoop.security.AccessControlException: 
>> Client cannot authenticate via:[TOKEN, KERBEROS]
>> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
>> at java.base/java.security.AccessController.doPrivileged(Native Method)
>> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
>> at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
>> at 
>> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
>> at 
>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
>> at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1389)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>> at 
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>> at 
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>> at 
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
>> at 
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
>> Method)
>> at 
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
>> Source)
>> at 
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
>> Source)
>> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
>> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
>> at 
>> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
>> at 
>> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
>> at 
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at 
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
>> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
>> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
>> at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
>> at

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Bjørn Jørgensen 
What is the JIRA ticket for this problem?

SPARK-25355  was marked
as resolved for 2 years ago. But now there are a lot of new comments on
whether things work and not.

SPARK-39033 Support --proxy-user for Spark on K8s not working
 is this
yours problem?

By having a good JIRA where the problem is discussed and the same with the
PR description helps when others try to change the code in the future.


tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar :

> Hi Steve
>
> Thx for the input  . Actually I have wrongly put the error stack
> trace(complete error trace and other details are available in the Jira ,
> apologies for the same. )  .  The error is related to authenticatication
> and it's happening only when the proxy user option is being used ,
> otherwise things are working fine.
>
> Also my intention wasn't to point out any mistakes or escalate the problem
> to the community.   I was just looking for help and direction to solve the
> issue.
>
> 22/04/26 08:54:40 DEBUG Client: closing ipc connection to /:8020: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]
> java.io.IOException: org.apache.hadoop.security.AccessControlException: 
> Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at 
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
> at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
> at org.apache.hadoop.ipc.Client.call(Client.java:1389)
> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source)
> at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
> at 
> org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
> at 
> org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
> at 
> org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
> at 
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
> at 
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at 
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
> at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
> at

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Pralabh Kumar 
Hi Steve

Thx for the input  . Actually I have wrongly put the error stack
trace(complete error trace and other details are available in the Jira ,
apologies for the same. )  .  The error is related to authenticatication
and it's happening only when the proxy user option is being used ,
otherwise things are working fine.

Also my intention wasn't to point out any mistakes or escalate the problem
to the community.   I was just looking for help and direction to solve the
issue.

22/04/26 08:54:40 DEBUG Client: closing ipc connection to
/:8020: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
at 
org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
at 
org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
at 
org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
at 
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at 
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at 
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at 
org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
at 
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364)
at scala.Option.map(Option.scala:230)
at 
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364)
at 
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Steve Loughran 
Prablah, did you follow the URL provided in the exception message? i put a
lot of effort in to improving the diagnostics, where the wiki articles are
part of the troubleshooing process
https://issues.apache.org/jira/browse/HADOOP-7469

it's really disappointing when people escalate the problem to open source
developers before trying to fix the problem themselves, in this case, read
the error message.

now, if there is some k8s related issue which makes this more common, you
are encouraged to update the wiki entry with a new cause. documentation is
an important contribution to open source projects, and if you have
discovered a new way to recreate the failure, it would be welcome. which
reminds me, i have to add something to connection reset and docker which
comes down to "turn off http keepalive in maven builds"

-Steve





On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi 
wrote:

> Hi,
>
> Please be aware that ConnectionRefused exception is has nothing to do w/
> authentication. See the description from Hadoop wiki:
> "You get a ConnectionRefused
>  
> Exception
> when there is a machine at the address specified, but there is no program
> listening on the specific TCP port the client is using -and there is no
> firewall in the way silently dropping TCP connection requests. If you do
> not know what a TCP connection request is, please consult the
> specification ."
>
> This means the namenode on host:port is not reachable in the TCP layer.
> Maybe there are multiple issues but I'm pretty sure that something is wrong
> in the K8S net config.
>
> BR,
> G
>
>
> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar 
> wrote:
>
>> Hi dev Team
>>
>> Spark-25355 added the functionality of the proxy user on K8s . However
>> proxy user on K8s with Kerberized HDFS is not working .  It is throwing
>> exception and
>>
>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
>> the server : org.apache.hadoop.security.AccessControlException: Client
>> cannot authenticate via:[TOKEN, KERBEROS]
>>
>>
>> Exception in thread "main" java.net.ConnectException: Call From
>>  to  failed on connection exception:
>> java.net.ConnectException: Connection refused; For more details see:  http:
>> //wiki.apache.org/hadoop/ConnectionRefused
>>
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>> Source)
>>
>> at
>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>> Source)
>>
>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
>>
>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>
>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>
>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>
>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>
>> at
>>
>>
>>
>> On debugging deep , we found the proxy user doesn't have access to
>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>> the proxy user and this user doesn't have delegation token.
>>
>>
>> Please help me with the same.
>>
>>
>> Regards
>>
>> Pralabh Kumar
>>
>>
>>
>>

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-04-30 Thread Gabor Somogyi 
Hi,

Please be aware that ConnectionRefused exception is has nothing to do w/
authentication. See the description from Hadoop wiki:
"You get a ConnectionRefused

Exception
when there is a machine at the address specified, but there is no program
listening on the specific TCP port the client is using -and there is no
firewall in the way silently dropping TCP connection requests. If you do
not know what a TCP connection request is, please consult the specification
."

This means the namenode on host:port is not reachable in the TCP layer.
Maybe there are multiple issues but I'm pretty sure that something is wrong
in the K8S net config.

BR,
G


On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar 
wrote:

> Hi dev Team
>
> Spark-25355 added the functionality of the proxy user on K8s . However
> proxy user on K8s with Kerberized HDFS is not working .  It is throwing
> exception and
>
> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
> the server : org.apache.hadoop.security.AccessControlException: Client
> cannot authenticate via:[TOKEN, KERBEROS]
>
>
> Exception in thread "main" java.net.ConnectException: Call From
>  to  failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:  http:
> //wiki.apache.org/hadoop/ConnectionRefused
>
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
> Source)
>
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
> Source)
>
> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
>
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>
> at
>
>
>
> On debugging deep , we found the proxy user doesn't have access to
> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
> the proxy user and this user doesn't have delegation token.
>
>
> Please help me with the same.
>
>
> Regards
>
> Pralabh Kumar
>
>
>
>

Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-04-29 Thread Pralabh Kumar 
Hi dev Team

Spark-25355 added the functionality of the proxy user on K8s . However
proxy user on K8s with Kerberized HDFS is not working .  It is throwing
exception and

22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]


Exception in thread "main" java.net.ConnectException: Call From 
to  failed on connection exception: java.net.ConnectException:
Connection refused; For more details see:  http://
wiki.apache.org/hadoop/ConnectionRefused

at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)

at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)

at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)

at org.apache.hadoop.ipc.Client.call(Client.java:1443)

at org.apache.hadoop.ipc.Client.call(Client.java:1353)

at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)

at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)

at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)

at



On debugging deep , we found the proxy user doesn't have access to
delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
the proxy user and this user doesn't have delegation token.


Please help me with the same.


Regards

Pralabh Kumar