dongjoon-hyun opened a new pull request, #727: URL: https://github.com/apache/spark-kubernetes-operator/pull/727
### What changes were proposed in this pull request? This PR excludes Hadoop YARN classes (`org/apache/hadoop/yarn/**`) from the operator shadow jar by adding a class-path filter to the `shadowJar` block in `spark-operator/build.gradle`. Note that the individual `org.apache.hadoop:hadoop-yarn-api`, `hadoop-yarn-client`, and `hadoop-yarn-common` artifacts are not present in the dependency graph. Spark's `spark-core` / `spark-kubernetes` pull in the shaded `hadoop-client-api` uber-jar, which bundles the YARN classes internally. Therefore these classes are filtered at packaging time rather than excluded as Maven modules. This follows the existing precedent of excluding Hadoop `webapps/**` (HDFS/YARN UI). ### Why are the changes needed? The operator manages Spark applications on Kubernetes and never uses YARN. The bundled YARN classes are dead weight in the runtime classpath and shaded jar. Removing them drops 2,766 classes (~20 MB uncompressed) that originate from `hadoop-client-api`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Additionally verified locally: - After the change, `org/apache/hadoop/yarn/` entries in the shaded jar dropped from 2,766 to 0, while other Hadoop classes (e.g. `org/apache/hadoop/thirdparty/**`) remain intact. - `:spark-operator:test` and `:spark-submission-worker:test` pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
