sarutak opened a new pull request, #58737:
URL: https://github.com/apache/spark/pull/58737
### What changes were proposed in this pull request?
Follow-up to SPARK-59296. Make OIDC credential propagation active in local
mode, for parity with `HadoopDelegationTokenManager` (which already runs in
`LocalSchedulerBackend` via `createTokenManager()`).
- `LocalSchedulerBackend` now starts a `UserCredentialManager` when
`spark.security.oidc.enabled=true`, mirroring `CoarseGrainedSchedulerBackend`.
It is started in `start()` (after the token manager) and stopped in `stop()`.
Both the token manager and the user-credential manager are stopped
independently (each wrapped in `Utils.tryLogNonFatalError`) so a failure in one
does not skip the other, and the renewal thread is shut down before
`SparkContext.stop()` closes the shared `CredentialProviderLoader`. In local
mode the driver and the single in-JVM executor share
`SparkEnv.get.userCredentials`, so the propagation callback updates that store
directly (there is no remote executor to message); tasks pick the credentials
up via `TaskDescription`.
- The provider **selection** phase
(`UserCredentialManager.applyProviderProperties`) now runs in local mode too.
SPARK-59296 skipped it in local mode because no resolution phase followed
there; now that `LocalSchedulerBackend` runs a resolution phase, the wiring it
applies to the driver's Hadoop `Configuration` points at credentials that are
actually populated. The `isLocal` parameter of `applyProviderProperties` (whose
only purpose was that skip) is removed; the `SparkContext` call site and the
surrounding scaladoc/comments are updated accordingly.
### Why are the changes needed?
Before this change, `spark.security.oidc.enabled=true` had no effect in
local mode: no `UserCredentialManager` was started, so OIDC credentials were
never acquired, and the selection phase was skipped so no provider wiring was
applied. `HadoopDelegationTokenManager`, by contrast, runs in
`LocalSchedulerBackend`, so Kerberos-based credential acquisition already works
in local mode. This closes that parity gap, so a local-mode driver acquires and
uses OIDC-derived
credentials for its own storage access (and renews them).
### Does this PR introduce _any_ user-facing change?
Yes (behavior only; no public API added or removed, and the feature is
unreleased). With OIDC enabled in local mode, the driver now acquires OIDC
credentials and applies provider-declared properties, instead of the feature
being a no-op. As in cluster mode, initial credential acquisition is fail-fast:
if OIDC is enabled but the identity token file is missing or malformed,
`SparkContext` startup fails rather than running with no credentials.
### How was this patch tested?
- New `LocalSchedulerBackendSuite` (real `SparkContext` in `local[1]`): with
OIDC enabled, the scheduler backend is a `LocalSchedulerBackend`, the selection
phase wires the provider-declared properties into the driver's Hadoop
`Configuration` (and non-Hadoop `spark.*` too), a loader is retained on
`SparkContext`, and the resolution phase populates
`SparkEnv.get.userCredentials`; with OIDC disabled, all of these are
no-ops.
- New `UserCredentialManagerSuite` test proving the selection -> resolution
ordering invariant: the selection phase does not initialize the provider
(`getInitCount` is 0 after `selectProviderForProperties`), and reusing the same
loader for resolution initializes the provider exactly once (`getInitCount`
becomes 1, and the loader returns the same cached
instance). The "no-op in local mode" test is replaced by one asserting
selection now applies properties regardless of local mode.
- `UserCredentialManagerSuite`, `OidcCredentialIntegrationSuite`, and
`LocalSchedulerBackendSuite` pass (49 tests). `SparkContextSuite` passes (84
tests) as a regression check on the modified `SparkContext` init path.
`dev/lint-scala` and `dev/lint-java` pass.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
