Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Gabor Somogyi 
If you take a look at the jiras then ..33 is the newly opened same issue
which is discussed on ..55.
The quality of the logs are the exact same which is useless for real
analysis...

Not going too far spark-submit has cluster mode, the logs are partial and
contains client mode (so they don't belong together).

> By having a good JIRA where the problem is discussed and the same with
the PR description helps when others try to change the code in the future.


Not sure what you mean here. The guys who made the changes understands the
feature/change which is not necessarily true for all others.
This is double true for security features which is super hairy area...

G


On Tue, May 3, 2022 at 9:28 PM Bjørn Jørgensen 
wrote:

> What is the JIRA ticket for this problem?
>
> SPARK-25355  was
> marked as resolved for 2 years ago. But now there are a lot of new comments
> on whether things work and not.
>
> SPARK-39033 Support --proxy-user for Spark on K8s not working
>  is
> this yours problem?
>
> By having a good JIRA where the problem is discussed and the same with the
> PR description helps when others try to change the code in the future.
>
>
> tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar :
>
>> Hi Steve
>>
>> Thx for the input  . Actually I have wrongly put the error stack
>> trace(complete error trace and other details are available in the Jira ,
>> apologies for the same. )  .  The error is related to authenticatication
>> and it's happening only when the proxy user option is being used ,
>> otherwise things are working fine.
>>
>> Also my intention wasn't to point out any mistakes or
>> escalate the problem to the community.   I was just looking for help and
>> direction to solve the issue.
>>
>> 22/04/26 08:54:40 DEBUG Client: closing ipc connection to 
>> /:8020: org.apache.hadoop.security.AccessControlException: 
>> Client cannot authenticate via:[TOKEN, KERBEROS]
>> java.io.IOException: org.apache.hadoop.security.AccessControlException: 
>> Client cannot authenticate via:[TOKEN, KERBEROS]
>> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
>> at java.base/java.security.AccessController.doPrivileged(Native Method)
>> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
>> at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
>> at 
>> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
>> at 
>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
>> at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1389)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>> at 
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>> at 
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>> at 
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
>> at 
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
>> Method)
>> at 
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
>> Source)
>> at 
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
>> Source)
>> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
>> at 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
>> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
>> at 
>> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
>> at 
>> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
>> at 
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at 
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
>> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
>> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
>> at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
>> at

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Bjørn Jørgensen 
What is the JIRA ticket for this problem?

SPARK-25355  was marked
as resolved for 2 years ago. But now there are a lot of new comments on
whether things work and not.

SPARK-39033 Support --proxy-user for Spark on K8s not working
 is this
yours problem?

By having a good JIRA where the problem is discussed and the same with the
PR description helps when others try to change the code in the future.


tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar :

> Hi Steve
>
> Thx for the input  . Actually I have wrongly put the error stack
> trace(complete error trace and other details are available in the Jira ,
> apologies for the same. )  .  The error is related to authenticatication
> and it's happening only when the proxy user option is being used ,
> otherwise things are working fine.
>
> Also my intention wasn't to point out any mistakes or escalate the problem
> to the community.   I was just looking for help and direction to solve the
> issue.
>
> 22/04/26 08:54:40 DEBUG Client: closing ipc connection to /:8020: 
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate 
> via:[TOKEN, KERBEROS]
> java.io.IOException: org.apache.hadoop.security.AccessControlException: 
> Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at 
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
> at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
> at org.apache.hadoop.ipc.Client.call(Client.java:1389)
> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown 
> Source)
> at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown 
> Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
> at 
> org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
> at 
> org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
> at 
> org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
> at 
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
> at 
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at 
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
> at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
> at

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Pralabh Kumar 
Hi Steve

Thx for the input  . Actually I have wrongly put the error stack
trace(complete error trace and other details are available in the Jira ,
apologies for the same. )  .  The error is related to authenticatication
and it's happening only when the proxy user option is being used ,
otherwise things are working fine.

Also my intention wasn't to point out any mistakes or escalate the problem
to the community.   I was just looking for help and direction to solve the
issue.

22/04/26 08:54:40 DEBUG Client: closing ipc connection to
/:8020: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at 
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
at 
org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
at 
org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
at 
org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
at 
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at 
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at 
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at 
org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
at 
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364)
at scala.Option.map(Option.scala:230)
at 
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364)
at 
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at

Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355

2022-05-03 Thread Steve Loughran 
Prablah, did you follow the URL provided in the exception message? i put a
lot of effort in to improving the diagnostics, where the wiki articles are
part of the troubleshooing process
https://issues.apache.org/jira/browse/HADOOP-7469

it's really disappointing when people escalate the problem to open source
developers before trying to fix the problem themselves, in this case, read
the error message.

now, if there is some k8s related issue which makes this more common, you
are encouraged to update the wiki entry with a new cause. documentation is
an important contribution to open source projects, and if you have
discovered a new way to recreate the failure, it would be welcome. which
reminds me, i have to add something to connection reset and docker which
comes down to "turn off http keepalive in maven builds"

-Steve





On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi 
wrote:

> Hi,
>
> Please be aware that ConnectionRefused exception is has nothing to do w/
> authentication. See the description from Hadoop wiki:
> "You get a ConnectionRefused
>  
> Exception
> when there is a machine at the address specified, but there is no program
> listening on the specific TCP port the client is using -and there is no
> firewall in the way silently dropping TCP connection requests. If you do
> not know what a TCP connection request is, please consult the
> specification ."
>
> This means the namenode on host:port is not reachable in the TCP layer.
> Maybe there are multiple issues but I'm pretty sure that something is wrong
> in the K8S net config.
>
> BR,
> G
>
>
> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar 
> wrote:
>
>> Hi dev Team
>>
>> Spark-25355 added the functionality of the proxy user on K8s . However
>> proxy user on K8s with Kerberized HDFS is not working .  It is throwing
>> exception and
>>
>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
>> the server : org.apache.hadoop.security.AccessControlException: Client
>> cannot authenticate via:[TOKEN, KERBEROS]
>>
>>
>> Exception in thread "main" java.net.ConnectException: Call From
>>  to  failed on connection exception:
>> java.net.ConnectException: Connection refused; For more details see:  http:
>> //wiki.apache.org/hadoop/ConnectionRefused
>>
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>> Source)
>>
>> at
>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>> Source)
>>
>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
>>
>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>
>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>
>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>
>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>
>> at
>>
>>
>>
>> On debugging deep , we found the proxy user doesn't have access to
>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>> the proxy user and this user doesn't have delegation token.
>>
>>
>> Please help me with the same.
>>
>>
>> Regards
>>
>> Pralabh Kumar
>>
>>
>>
>>

REMINDER - Travel Assistance available for ApacheCon NA New Orleans 2022

2022-05-03 Thread Gavin McDonald 
Hi All Contributors and Committers,

This is a first reminder email that travel
assistance applications for ApacheCon NA 2022 are now open!

We will be supporting ApacheCon North America in New Orleans, Louisiana,
on October 3rd through 6th, 2022.

TAC exists to help those that would like to attend ApacheCon events, but
are unable to do so for financial reasons. This year, We are supporting
both committers and non-committers involved with projects at the
Apache Software Foundation, or open source projects in general.

For more info on this year's applications and qualifying criteria, please
visit the TAC website at http://www.apache.org/travel/
Applications are open and will close on the 1st of July 2022.

Important: Applicants have until the closing date above to submit their
applications (which should contain as much supporting material as required
to efficiently and accurately process their request), this will enable TAC
to announce successful awards shortly afterwards.

As usual, TAC expects to deal with a range of applications from a diverse
range of backgrounds. We therefore encourage (as always) anyone thinking
about sending in an application to do so ASAP.

Why should you attend as a TAC recipient? We encourage you to read stories
from
past recipients at https://apache.org/travel/stories/ . Also note that
previous TAC recipients have gone on to become Committers, PMC Members, ASF
Members, Directors of the ASF Board and Infrastructure Staff members.
Others have gone from Committer to full time Open Source Developers!

How far can you go! - Let TAC help get you there.