voonhous commented on issue #20384: URL: https://github.com/apache/hudi/issues/20384#issuecomment-6314584346
Comparison of what Azure runs vs GitHub Actions, as of master `7ad42f1c7ad6` ([azure-pipelines-20230430.yml](https://github.com/apache/hudi/blob/7ad42f1c7ad6/azure-pipelines-20230430.yml), [bot.yml](https://github.com/apache/hudi/blob/7ad42f1c7ad6/.github/workflows/bot.yml)). Azure runs everything on Spark 3.5 / Scala 2.12 / Flink 2.2 / JDK 11. | Test area | Azure | GitHub Actions | |---|---|---| | hudi-hadoop-common UT, spark-client UT+FT | Spark 3.5 | Spark 3.5 (same) | | hudi-cli UT+FT | Spark 3.5 | Spark 3.5 (same) | | hudi-utilities UT+FT | Spark 3.5 | Spark 3.5 (same) | | Common and other modules UT+FT (incl. hudi-flink UT) | Spark 3.5 | Spark 3.5 (same) | | hudi-hadoop-mr, hudi-java-client UT | not run (FT only) | Spark 3.5 | | Spark datasource Java UT (hudi-spark, hudi-spark-common) | **Spark 3.5 / 2.12** | Spark 4.2 / 2.13 only | | Spark SQL Scala suites (dml, ddl, others, feature) | **Spark 3.5 / 2.12** | Spark 4.2 / 2.13 only | | hudi-spark functional tests A/B/C | **Spark 3.5 / 2.12** | Spark 4.2 / 2.13 only | | hudi-spark3.5.x + hudi-spark3-common module tests | **yes** | not run | | hudi-spark4-common + hudi-spark4.2.x module tests | not run | yes | | hudi-examples-spark UT | not run | Spark 4.2 | | Flink quickstart + hudi-flink ITs | not run | Flink 2.2 | | Docker integration tests (hudi-integ-test) | not run | Spark 3.5 | | Bundle validation | not run | Spark 3.5 + Flink 2.2 (JDK 11/17), Spark 4.2 (JDK 17) | | Coverage report | Azure JaCoCo | Codecov | What GitHub Actions is missing is the bold rows: the Spark datasource suites on Spark 3.5 / Scala 2.12, and the hudi-spark3.5.x / hudi-spark3-common adapter tests. The other four Azure test jobs already run on GitHub Actions with the same profile. To bring them over, the six Spark datasource jobs in bot.yml (`test-spark-java17-java-tests-part1/2/3`, `test-spark-java17-scala-dml-tests`, `test-spark-java17-scala-other-tests`) would need a `scala-2.12` / `spark3.5` matrix entry on JDK 11. The old Spark 3.x entries are still there, commented out under `[CI-TRIM]`. These jobs hard-code JDK 17 and `-Pjava17`, so both would need to come from the matrix. That adds six jobs per PR. Neither CI runs tests on Spark 3.3, 3.4, 4.0 or 4.1, and neither builds Flink 1.18 to 2.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
