Parth Chandra created SPARK-58329:
-------------------------------------
Summary: Enable direct credential providers on standalone clusters
Key: SPARK-58329
URL: https://issues.apache.org/jira/browse/SPARK-58329
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.3.0
Reporter: Parth Chandra
[SPARK-38954|https://issues.apache.org/jira/browse/SPARK-38954] added
{{spark.security.credentials.directProviders.enabled}}, which lets
{{HadoopDelegationTokenProvider}} implementations distribute credentials
without Kerberos. The feature is supported on YARN, Kubernetes, and local mode
but not on standalone clusters.
{{StandaloneSchedulerBackend}} inherits {{SupportsDelegationToken}} via
{{CoarseGrainedSchedulerBackend}} but does not override
{{createTokenManager()}}, so the default {{None}} is returned and the feature
is silently inert. A warning is logged when the config is set on standalone.
All required infrastructure is already in place:
* {{CoarseGrainedSchedulerBackend.DriverEndpoint.receive()}} handles
{{UpdateDelegationTokens}} and broadcasts tokens to executors via
{{executorDataMap}}.
* {{CoarseGrainedExecutorBackend.receive()}} applies received tokens via
{{SparkHadoopUtil.addDelegationTokens}}.
* {{driverEndpoint}} is available as the {{RpcEndpointRef}} required by
{{HadoopDelegationTokenManager}}.
The fix is a single method override in {{StandaloneSchedulerBackend}}:
{code:scala}
override protected def createTokenManager():
Option[HadoopDelegationTokenManager] = {
Some(new HadoopDelegationTokenManager(conf, sc.hadoopConfiguration,
driverEndpoint))
}
{code}
One detail to verify: {{setupTokenManager()}} is called at the start of
{{CoarseGrainedSchedulerBackend.start()}}, before {{driverEndpoint}} is
initialised. YARN and K8s set up their endpoints before calling
{{super.start()}}. {{StandaloneSchedulerBackend.start()}} calls
{{super.start()}} first, so the call
ordering may need adjustment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
