dongjoon-hyun opened a new pull request, #988: URL: https://github.com/apache/spark-kubernetes-operator/pull/988
### What changes were proposed in this pull request? This PR aims to upgrade Apache Spark to 4.3.0. Docs, examples, E2E tests and the default image of `SparkClusterResourceSpec` are not changed, because the `apache/spark:4.3.0` Docker images are not published yet. ### Why are the changes needed? Apache Spark 4.3.0 is released officially (announced on 2026-10-30). - https://github.com/apache/spark/releases/tag/v4.3.0 (tag created on 2026-10-27) - https://spark.apache.org/releases/spark-release-4-3-0.html - https://spark.apache.org/docs/4.3.0/ - https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.13/4.3.0/ Among the resolved runtime and test dependencies, only the Spark modules change, in addition to these new artifacts: - `org.apache.spark:spark-config_2.13`, which also bundles a relocated `protobuf-java` - `org.apache.spark:spark-udf-worker-core_2.13` - `org.apache.spark:spark-udf-worker-proto_2.13` - `org.spark-project.spark:unused`, which is not bundled in the shadow jar `LICENSE-binary` lists the bundled ones, and `gradle/verification-metadata.xml` is regenerated. The shadow jar grows from 75.7 MB to 79.2 MB, since `SparkEnv` now refers to the UDF worker classes, which keeps them and `protobuf-java` in the minimized jar. ### Does this PR introduce _any_ user-facing change? Yes. The operator builds the driver pod with the Spark libraries, whatever `spec.runtimeVersions.sparkVersion` is. `docs/migration_guide.md` describes the following changes and how to restore the previous behavior. - The driver container takes `allowPrivilegeEscalation` from `spark.kubernetes.driver.securityContext.allowPrivilegeEscalation`, `false` by default, which overrides a `true` in the driver pod template. - `SPARK_USER` of the driver container is `spec.proxyUser` if it is set, instead of the user of the operator process. The driver Service also has `publishNotReadyAddresses: false` explicitly now, which is the Kubernetes default and has no effect. ### How was this patch tested? Pass the CIs. I also ran the following locally with the regenerated `verification-metadata.xml`. - The full `gradle build` (unit tests, Checkstyle, PMD, SpotBugs, Spotless and `checkLicenseBinary`) and `gradle javadoc` - `gradle test -PtestJavaVersion=21` - A comparison of the driver resources rendered with Spark 4.2.1 and 4.3.0, using a temporary test which is not included in this PR ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
