nsivabalan commented on code in PR #20453: URL: https://github.com/apache/hudi/pull/20453#discussion_r4405763921
########## .github/workflows/bot.yml: ########## @@ -65,6 +65,55 @@ env: SCALA_TEST_OTHERS_FILTER: -DwildcardSuites=org.apache.hudi,org.apache.spark.hudi,org.apache.spark.sql.avro,org.apache.spark.sql.execution,org.apache.spark.sql.hive,org.apache.spark.sql.hudi.analysis,org.apache.spark.sql.hudi.blob,org.apache.spark.sql.hudi.catalog,org.apache.spark.sql.hudi.command,org.apache.spark.sql.hudi.common,org.apache.spark.sql.hudi.ddl,org.apache.spark.sql.hudi.procedure,org.apache.spark.sql.hudi.feature jobs: + # Spark 4.1 entries in the general test matrices below are disabled by CI-TRIM. + # Keep SDP's pinned 4.1.1 client/server and candidate bundle in one focused lane. + test-spark-connect-pipelines: Review Comment: This lane runs on every PR and push that changes any non-doc file, and each run builds the Spark 4.1 bundle with `-am` and installs PySpark (~400 MB) before the smoke test. That brings back a good part of the Spark 4.1 cost that CI-TRIM removed, including for PRs that cannot affect SDP (Flink, Trino, docs-adjacent changes). Could we scope it, e.g. move it to its own workflow with a `paths:` filter (`hudi-spark-datasource/**`, `hudi-client/**`, `hudi-common/**`, `packaging/hudi-spark-bundle/**`, `hudi-integ-test/spark-pipelines/**`, the workflow file), and/or run it on push to master plus a nightly schedule? Also worth checking that 40 minutes is enough for a cold-cache `-am` bundle build on `ubuntu-24.04`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
