sarutak commented on code in PR #57998:
URL: https://github.com/apache/spark/pull/57998#discussion_r3783825361
##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -1089,6 +1089,15 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
// is processed by DriverEndpoint.
VersionedCredentials.updateIfNewer(
SparkEnv.get.userCredentials, version, initialCredentials)
+
+ // Auto-configure the executor-side S3A credentials provider if the user
has not
+ // explicitly set one. This ensures S3A reads dynamic credentials from
the executor
+ // store without requiring manual Hadoop configuration.
+ val s3aProviderKey = "spark.hadoop.fs.s3a.aws.credentials.provider"
Review Comment:
I traced the propagation path and confirmed this works correctly for
executors:
- `sparkProperties` (used in `SparkAppConfig` reply) is a `lazy val` on
`conf.getAll`, evaluated after `setupUserCredentialManager` completes -> OK
- Executor's `SparkHadoopUtil.newConfiguration(conf)` picks up
`spark.hadoop.*` from the received conf -> OK
However, two questions:
**(a) Driver-side S3A:** `SparkContext._hadoopConfiguration` is created (in
`SparkContext` constructor, via `SparkHadoopUtil.get.newConfiguration(_conf)`)
before `TaskSchedulerImpl.start()` triggers `setupUserCredentialManager`. This
means the driver's own Hadoop Configuration does not pick up this late
mutation. Is this intentional?
- If yes (executor-only by design): please add a comment explaining that
this setting only takes effect on executors because the driver's Hadoop
Configuration is already materialized.
- If no (driver should also use this provider): the auto-config needs to be
moved earlier, or the driver needs to refresh its Hadoop Configuration after
setup.
**(b) Classpath guard:** The `credential-aws` module is activated by an
optional Maven/SBT profile (`-Pcredential-aws`). If a user enables OIDC but
doesn't include the profile, this sets a class name that doesn't exist on the
classpath. The failure won't surface until the first S3A access (a
`ClassNotFoundException` inside Hadoop's reflection-based provider
instantiation). Please consider adding:
```scala
val providerClass =
"org.apache.spark.security.aws.SparkOidcAwsCredentialsProvider"
if (!conf.contains(s3aProviderKey)) {
try {
Utils.classForName(providerClass)
conf.set(s3aProviderKey, providerClass)
} catch {
case _: ClassNotFoundException =>
logWarning(log"credential-aws module not on classpath; " +
log"skipping S3A provider auto-configuration")
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
