sarutak commented on PR #57675:
URL: https://github.com/apache/spark/pull/57675#issuecomment-5176155844
Thank you, @dongjoon-hyun.
> 1. Stale-credential overwrite race in Executor (main comment)
Added a monotonic version counter in `UserCredentialManager` (incremented on
each renewal). The executor credential store is now
`AtomicReference[VersionedCredentials]` where `VersionedCredentials(version:
Long, bytes: Array[Byte])`
. Both `Executor.TaskRunner` and `CoarseGrainedExecutorBackend.receive` use
`AtomicReference.updateAndGet` to atomically apply credentials only if the
incoming version is strictly greater than the current one:
```scala
env.userCredentials.updateAndGet { current =>
if (current == null || version > current.version) newValue else current
}
```
This should eliminate the race you described. A delayed TaskDescription
carrying v1 will not overwrite fresher v2 credentials delivered via RPC.
> 2. Redundant initial propagation
Added a comment in `setupUserCredentialManager()` explaining why both the
synchronous set and the async callback exist:
```
// Note: the onCredentialsUpdate callback above also triggers an async
// UpdateUserCredentials message that will redundantly set these stores.
// The synchronous set here ensures no null window before the async message
// is processed by DriverEndpoint.
```
> 3. Per-task payload overhead
Added a comment in `TaskDescription` documenting the trade-off:
```
// Trade-off: every TaskDescription carries the full serialized
UserCredentials (a few KB).
// This is acceptable because OIDC credentials are short-lived (minutes)
unlike Hadoop
// delegation tokens (hours/days), making the race between RPC broadcast and
task dispatch
// a practical concern. For short-task-heavy workloads, the overhead is
bounded by
// credential size × tasks-in-flight (not total task count).
```
> 4. Minor
- Fixed `SparkEnv.userCredentials` Scaladoc to say "on both driver and
executors".
- Added executor-side version guard unit test ("executor-side credential
store version guard rejects stale and accepts newer").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
