parthchandra opened a new pull request, #57285:
URL: https://github.com/apache/spark/pull/57285
### What changes were proposed in this pull request?
This PR enables Spark's existing credential distribution mechanism (driver
obtains tokens, pushes them to executors, refreshes before expiry) to work
without Kerberos. Today this mechanism is unconditionally gated
on Kerberos being enabled. This change adds a single configuration switch
(`spark.security.credentials.directProviders.enabled`) that lifts the Kerberos
gate on token collection.
Changes:
- `HadoopDelegationTokenManager.renewalEnabled` now also returns true when
the new config is enabled.
- `obtainDelegationTokens(creds)` and `obtainTokensAndScheduleRenewal()`
branch on whether Kerberos credentials are present: if yes, call providers
inside `doLogin()`/`doAs()` as before; if not (but the switch is
on), call providers directly. Providers that require Kerberos self-gate
via `delegationTokensRequired` returning `false`.
- Per-provider try/catch for failure isolation — one provider throwing
does not block others.
- `SupportsDelegationToken` gains an overridable `tokenManagerRequired()`
hook so scheduler backends can activate the token manager based on the new
config.
- Hard startup failure if `spark.network.crypto.enabled` is not true when
the switch is active (prevents bearer tokens over plaintext RPC).
- WARN log when the config is set but no providers are discovered via
ServiceLoader.
Design document: [SPIP: Cloud Credential Refresh and Distribution Without
Kerberos](https://issues.apache.org/jira/browse/SPARK-38954)
### Why are the changes needed?
In non-Kerberos deployments (cloud, internal IdP, cross-account access on
YARN), each executor independently authenticates against identity services,
causing thundering-herd load amplification, inconsistent
credential state, and no centralized control over credential rotation. The
existing distribution infrastructure is already provider-agnostic (Kafka proves
this) — only the activation gates prevent it from running
without Kerberos.
### Does this PR introduce _any_ user-facing change?
Yes. New configuration:
| Config | Default | Description |
|--------|---------|-------------|
| `spark.security.credentials.directProviders.enabled` | `false` | Enables
delegation token collection and renewal without Kerberos. When true, the
manager starts even if Hadoop security is not enabled, calling all
providers whose `delegationTokensRequired` returns true. Requires
`spark.network.crypto.enabled=true`. |
Existing deployments are unaffected (config defaults to false). When
Kerberos IS present, all code paths are unchanged.
### How was this patch tested?
New unit test suite `NonKerberosCredentialsSuite` (7 tests):
- `renewalEnabled` returns true/false based on config
- Providers are called without Kerberos when config is enabled
- Providers with `delegationTokensRequired=false` are skipped
- A failing provider does not prevent other providers from running
- Individual provider can be disabled via
`spark.security.credentials.<service>.enabled`
- Startup fails if `spark.network.crypto.enabled` is not true
Existing `HadoopDelegationTokenManagerSuite` (4 tests) passes unmodified —
verifies no regression in Kerberos deployments.
### Was this patch authored or co-authored using generative AI tooling?
Yes, Co-authored by Claude Code (opus 4.6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
