sarutak opened a new pull request, #55218: URL: https://github.com/apache/spark/pull/55218
### What changes were proposed in this pull request? This PR is a part of SPARK-56352 for Scala-based test JARs, replacing pre-built test JAR files containing Scala classes with dynamic compilation at test time, removing 6 binary JAR files and 1 binary from the repository. Changes: - Add `TestUtils.createJarWithScalaSources()` in `SparkTestUtils.scala` that compiles Scala source files via `scala.tools.nsc.Main` and packages the resulting classes into a JAR, with support for excluding specific classes by prefix. - Update test suites to use dynamically generated JARs instead of pre-built ones. - Refactor `StubClassLoaderSuite` to use a self-contained dummy class instead of the pre-built `udf_noA.jar` that contained spark-connect classes, eliminating the cross-module dependency noted in the original TODO comment. - Extract `StubClassDummyUdfPacker` from `StubClassDummyUdf.scala` into a separate file for use by `UDFClassLoadingE2ESuite`. - Remove deleted JAR/binary entries from `dev/test-jars.txt`. JARs/binaries removed: - `core/src/test/resources/TestHelloV2_2.13.jar` - `core/src/test/resources/TestHelloV3_2.13.jar` - `sql/connect/client/jvm/src/test/resources/TestHelloV2_2.13.jar` - `sql/connect/client/jvm/src/test/resources/udf2.13.jar` - `sql/connect/client/jvm/src/test/resources/udf2.13` (serialized binary) - `sql/core/src/test/resources/artifact-tests/udf_noA.jar` - `sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.13.jar` - ### Why are the changes needed? As noted in the PR discussion (https://github.com/apache/spark/pull/50378): > the ultimate goal is to refactor the tests to automatically build the jars > instead of using pre-built ones This PR achieves that goal for all Scala-based test JARs. By generating JARs dynamically at test time, no binary artifacts need to be stored in the source tree, and the release-time workaround becomes unnecessary for these files. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? All affected test suites pass: - ClassLoaderIsolationSuite (core) - StubClassLoaderSuite, ArtifactManagerSuite (sql/core) - HiveSparkSubmitSuite (sql/hive) - ReplE2ESuite (sql/connect) - UDFClassLoadingE2ESuite (sql/connect) ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Opus 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
