[
https://issues.apache.org/jira/browse/SPARK-38862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jack updated SPARK-38862:
-------------------------
Description:
[Spark documentation|https://spark.apache.org/docs/latest/security.html] states
that
??The REST Submission Server and the MesosClusterDispatcher do not support
authentication. You should ensure that all network access to the REST API &
MesosClusterDispatcher (port 6066 and 7077 respectively by default) are
restricted to hosts that are trusted to submit jobs.??
Whilst it is true that we can use network policies to restrict access to our
exposed submission endpoint, it would be preferable to at least also allow some
primitive form of authentication at a global level, whether this is by some
token provided to the runtime environment or is a "system user" using basic
authentication of a username/password combination - I am not strictly
opinionated and I think either would suffice.
Alternatively, one could implement a custom proxy to provide this
authentication check, but upon investigation this option is rejected by the
spark master as-is today.
I would imagine that whatever solution is agreed for a first phase, a custom
authenticator may be something we want a user to be able to provide so that if
an admin needed some more advanced authentication check, such as RBAC et al, it
could be facilitated without the need for writing a complete custom proxy
layer; although it could be argued just some basic built in layer being
available; eg. RestSubmissionBasicAuthenticator could be preferable.
was:
[Spark documentation|https://spark.apache.org/docs/latest/security.html] states
that
??The REST Submission Server and the MesosClusterDispatcher do not support
authentication. You should ensure that all network access to the REST API &
MesosClusterDispatcher (port 6066 and 7077 respectively by default) are
restricted to hosts that are trusted to submit jobs.??
Whilst it is true that we can use network policies to restrict access to our
exposed submission endpoint, it would be preferable to at least also allow some
primitive form of authentication at a global level, whether this is by some
token provided to the runtime environment or is a "system user" using basic
authentication of a username/password combination - I am not strictly
opinionated and I think either would suffice.
I appreciate that one could implement a custom proxy to provide this
authentication check, but it seems like a common use case that others may
benefit from to be able to authenticate against the rest submission endpoint,
and by implementing this capability as an optionally configurable aspect of
Spark itself, we can utilise the existing server to provide this check.
I would imagine that whatever solution is agreed for a first phase, a custom
authenticator may be something we want a user to be able to provide so that if
an admin needed some more advanced authentication check, such as RBAC et al, it
could be facilitated without the need for writing a complete custom proxy
layer; but I do feel there should be some basic built in available; eg.
RestSubmissionBasicAuthenticator.
> Let consumers provide their own method for Authentication for The REST
> Submission Server
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-38862
> URL: https://issues.apache.org/jira/browse/SPARK-38862
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core, Spark Submit
> Affects Versions: 3.4.0
> Reporter: Jack
> Priority: Major
> Labels: authentication, pull-request-available, rest, spark,
> spark-submit, submit
>
> [Spark documentation|https://spark.apache.org/docs/latest/security.html]
> states that
> ??The REST Submission Server and the MesosClusterDispatcher do not support
> authentication. You should ensure that all network access to the REST API &
> MesosClusterDispatcher (port 6066 and 7077 respectively by default) are
> restricted to hosts that are trusted to submit jobs.??
> Whilst it is true that we can use network policies to restrict access to our
> exposed submission endpoint, it would be preferable to at least also allow
> some primitive form of authentication at a global level, whether this is by
> some token provided to the runtime environment or is a "system user" using
> basic authentication of a username/password combination - I am not strictly
> opinionated and I think either would suffice.
> Alternatively, one could implement a custom proxy to provide this
> authentication check, but upon investigation this option is rejected by the
> spark master as-is today.
> I would imagine that whatever solution is agreed for a first phase, a custom
> authenticator may be something we want a user to be able to provide so that
> if an admin needed some more advanced authentication check, such as RBAC et
> al, it could be facilitated without the need for writing a complete custom
> proxy layer; although it could be argued just some basic built in layer being
> available; eg. RestSubmissionBasicAuthenticator could be preferable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
