sarutak opened a new pull request, #57679: URL: https://github.com/apache/spark/pull/57679
### What changes were proposed in this pull request? This PR creates the `connector/credential-aws` module scaffolding as a part of the OIDC Credential Propagation SPIP (SPARK-57703). **Module definition:** - `connector/credential-aws/pom.xml` (parent: spark-parent_2.13, deps: spark-core provided, spark-tags, junit-jupiter test) - Root `pom.xml`: `<profile><id>credential-aws</id>` with `<module>connector/credential-aws</module>` - `project/SparkBuild.scala`: added `credentialAws` to `optionallyEnabledProjects` and MiMa exclusion list - `dev/sparktestsupport/modules.py`: `credential_aws` Module definition - `dev/sparktestsupport/utils.py`: doctest expected output update **ServiceLoader registration and stub provider:** - `META-INF/services/org.apache.spark.security.CredentialProvider` pointing to `AwsStsCredentialProvider` - Stub `AwsStsCredentialProvider` — minimal implementation for ServiceLoader discovery testing. This class will be replaced by the full implementation in SPARK-57898 (Sub-task 9). The stub is included here to verify that the ServiceLoader wiring, test framework setup, and CI profile activation work end-to-end. **CI/release/lint script updates (`-Pcredential-aws` added in 22 locations):** - `.github/workflows/build_and_test.yml` (is-changed.py module list, SBT build, Maven build) - `.github/workflows/maven_test.yml` (all build/test commands) - `.github/workflows/python_hosted_runner_test.yml` - `.github/workflows/benchmark.yml` - `dev/create-release/release-build.sh` - `dev/test-dependencies.sh` - `dev/mima` - `dev/lint-java` - `dev/sbt-checkstyle` - `dev/scalastyle` - `docs/_plugins/build_api_docs.rb` - `dev/spark-test-image-util/docs/build-docs` **Note:** This PR establishes the build infrastructure only (stub provider). The actual provider implementation and its tests will follow in SPARK-57898, which will exercise the profile end-to-end (compilation, test execution, lint checks). ### Why are the changes needed? The AWS reference provider needs a separate module to keep AWS SDK dependencies out of Spark core. Splitting the module scaffolding from the implementation: 1. Makes the CI/release integration independently reviewable (12 scripts, 22 locations). 2. Allows SPARK-57898 and SPARK-57899 to focus purely on the provider implementation and tests after rebasing onto this PR. ### Does this PR introduce _any_ user-facing change? No. The module is activated only via `-Pcredential-aws` (Maven) or by including `credential-aws` in SBT's `optionallyEnabledProjects`. It is not included in the default build. The stub provider throws `UnsupportedOperationException` on `init()` and `resolve()`. ### How was this patch tested? - `mvn -Pcredential-aws -pl connector/credential-aws test`: 5 tests pass (ServiceLoader discovery, supportedSchemes, init throws, resolve throws, suggestedTtl) - `build/sbt -Pcredential-aws "credential-aws/test"`: 5 tests pass - `python3 -m doctest dev/sparktestsupport/utils.py`: passes with updated expected output - `./dev/lint-java`: Checkstyle checks passed - `./dev/mima`: success - `./dev/scalastyle`: Scalastyle checks passed - `mvn -Pcredential-aws -pl connector/credential-aws apache-rat:check`: all source files have valid license headers ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Claude -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
