andygrove opened a new pull request, #5885: URL: https://github.com/apache/datafusion-comet/pull/5885
## Which issue does this PR close? Part of #4329. This PR does **not** close it — removal is deferred, not cancelled, so #4329 stays open to track the eventual drop. It will need its `1.1.0` milestone moved. ## Rationale for this change Spark 3.4 was slated for removal in 1.1.0, and the docs said so in three places. We got pushback on that timing, so this PR walks the commitment back: Spark 3.4 becomes deprecated with no scheduled removal date, and we keep building and publishing its binaries. What deprecation buys us is CI time. The Spark 3.4 Spark SQL suite currently runs in the merge queue on every merge, where it is one of the slowest jobs and a flake in it blocks everyone. Since we are no longer committing to fix Spark 3.4 regressions with the same urgency, it does not need to gate merges. It does still need to be runnable. Anyone working on `spark/src/main/spark-3.4/` or `dev/diffs/3.4.3.diff` should be able to get a verdict before merging, and `run-spark-3.4-tests` already exists for exactly that. ## What changes are included in this PR? **CI.** `spark_3_4` in `dev/ci/compute-changes.py` goes from `["queue", "label:run-spark-3.4-tests"]` to `["label:run-spark-3.4-tests"]`, which drops it from the merge queue while leaving the label and `workflow_dispatch` paths intact: | Event | `spark_3_4` | | --- | --- | | `merge_group` | false | | PR, no label | false | | PR with `run-spark-3.4-tests` | true | | `labeled` + `run-spark-3.4-tests` | true | | `workflow_dispatch` | true | Nothing else moves tier — 3.5, 4.0, 4.1 and all four Iceberg jobs still run in the queue. `spark_3_4` stays in `required_checks.needs`, so a labelled run still gates the pull request it was asked for. **Docs.** - `compatibility/spark-versions.md`, `installation.md`: removal target changes from "the 1.1.0 release" to "a future release", and both now say the Spark SQL suite no longer runs on every change, so a Spark 3.4 regression is more likely to reach a release. The installation CI table reads `On demand` for 3.4's Spark SQL column. The JDK 11 deprecation was sharing a sentence with Spark 3.4 — I split them so its 1.1.0 removal is unaffected. - `about/versioning_policy.md`: the lifecycle said removal happens in "the following" Comet minor release, and gave Spark 3.4 / 1.1.0 as the worked example. Reworded to "a later Comet minor release", removal not scheduled in advance, with "stops gating merges on the Spark SQL suite" folded into what deprecation means. **This is the one edit that changes policy rather than just describing the new state** — happy to instead keep the next-minor promise and write 3.4 up as a stated exception, if reviewers prefer that. - `contributor-guide/ci.md`, `.github/workflows/README.md`, `ci.yml` header comments: 3.4 moves out of the queue-tier listings into its own "label or dispatch only" row, with guidance to reach for the label deliberately since nothing else will run it. - `contributor-guide/spark-sql-tests.md`: the "Running Tests in CI" section claimed opening a PR was enough and pointed at `spark_sql_test.yaml` / `spark_sql_test_ansi.yaml`, which no longer exist. Rewritten with the per-version reality and the `gh pr edit --add-label` command. ## How are these changes tested? `dev/ci/check-ci-config.py` is the test for the routing policy, and it needed updating in a way that is itself the coverage. `spark_3_4` moves out of `SPARK_OPT_IN` into its own `SPARK_DEPRECATED` set, which keeps it in `ALL_JOBS` (so `workflow_dispatch` still asserts it runs) while taking it out of `QUEUE_TIER` (so the `merge_group` case now asserts its *absence* rather than accepting it coming back silently). Added a `labeled` + `run-spark-3.4-tests` case to pin that the manual path stays reachable. Locally green: `check-ci-config.py`, `check-suites.py`, `test-iceberg-shards.py`, `apache-rat:check`, `actionlint --shellcheck=off`, and `prettier --check` on every file this PR touches. I also drove `compute-changes.py` directly for each event shape to produce the table above, rather than trusting the assertions alone. Three markdown files fail `prettier --check` at this branch's base (`.ai/skills/review-comet-pr/SKILL.md`, `contributor-guide/debugging.md`, `contributor-guide/native_shuffle.md`). Pre-existing and untouched here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
