Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin:
> @mridulm the main argument for just dealing with Hadoop security is that
it's been sufficient since the inception of Spark. I have never seen anyone ask
for integration with any other type of system.
Support for long running applications (which require token renewal, etc)
was added much later in spark - in 1.4 IIRC :
https://issues.apache.org/jira/browse/SPARK-5342
> Would it make you more comfortable if the new API were kept
private[spark]? It would limit extensibility in the case of Mesos (would be
restricted to built-in providers), but would free us from this discussion and
allow progress to happen.
If we are not exposing an api for spark core, while maintaining backward
compatibility - I am fine with the change. (Please see [1] below too)
Either we move implementations from yarn to core - or a new module which
yarn and mesos depends on (if that helps).
@mgummelt:
> @mridulm You keep mentioning hadoop-security as if it's a library. It's
not. UserGroupInformation and Credentials, for example, are are security
classes in hadoop-commons, which core already depends on. So this coupling
already exists. Is your concern that we're increasing this coupling?
When I refer to hadoop-security, I do not mean a maven package - but use of
`org.apache.hadoop.security` for handling authentication/authorization.
hadoop-common contains a collection of libraries & utilities, bundled together
for convenience; `security` package implements classes in context of security
design of hadoop.
[1] If we want to base security in spark on hadoop-security, and think it
is sufficient for our current & anticipated needs - we should be explicit
about the (design) dependency.
We should solicit opinion about it in dev@ and proceed based on feedback
from from the list (this PR discussion might not be followed by many interested
parties).
As @vanzin mentioned above, there have not integration requests for other
systems - so perhaps it is sufficient for our needs and I might be being overly
paranoid (due to my past experiences with api design).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
