Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16788
I took a very quick look at this; Mridul and Saisai have already raised
good questions about this.
But I'm a little worried that this is creating yet another way of dealing
with Kerberos that will lead to two different code paths that don't share much
with each other.
Spark-on-YARN's support for Kerberos is really nothing complicated. The
only thing that Spark needs from YARN is a method of distributing the
principal's credentials. YARN just happens to have an API for that (see
`setupSecurityToken` in Client.scala).
On the other side, YARN just writes the credentials to a file and sets
`HADOOP_TOKEN_FILE_LOCATION` in the container process's environment to point to
the file; `UserGroupInformation` reads that automatically, and things work.
So it seems to me that all that would be needed from Mesos is a way to
replicate that behavior (i.e. distribute these credentials securely). The rest
(including the code to deal with delegation tokens) could just be factored out
of the YARN module, which should be easy now that Spark depends on Hadoop 2.6.
Also, note that `--principal` and `--keytab` are *not* required to run a
Spark application in a kerberized cluster. Things should also work when you
just login using `kinit` and run the Spark app. `UserGroupInformation` already
handles that case for you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
