Shrirang Mhalgi created SPARK-58789:
---------------------------------------
Summary: Add CredentialProvider.additionalSparkProperties() SPI
for executor-side auto-configuration
Key: SPARK-58789
URL: https://issues.apache.org/jira/browse/SPARK-58789
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 4.4.0
Reporter: Shrirang Mhalgi
Add a default method `additionalSparkProperties()` to the `CredentialProvider`
interface that allows provider modules to declare Spark configuration
properties needed when the provider is active.
After successful credential acquisition, `UserCredentialManager.start()`
iterates all discovered providers and applies their declared properties to
`SparkConf` – only if the user has not already set them explicitly. This
enables zero-config executor-side wiring (e.g., S3A credentials provider)
without placing vendor-specific logic in core.
h4. Motivation:
- PR #57998 (SparkOidcAwsCredentialsProvider) originally placed auto-config
logic in CoarseGrainedSchedulerBackend, which was a layer violation (core
referencing an optional module's class)
- This PR moves the mechanism to a generic SPI where each provider module
declares its own properties
h4. Changes:
- CredentialProvider.java: new default method `additionalSparkProperties()`
(returns Map<String, String>, empty by default)
- CredentialProviderLoader.java: new `discoverAllProviders()` for querying
provider metadata
- UserCredentialManager.scala: apply provider properties after start()
- AwsStsCredentialProvider.java: override returning S3A executor provider
mapping
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
