andygrove opened a new pull request, #5939:
URL: https://github.com/apache/datafusion-comet/pull/5939
## Which issue does this PR close?
Closes #5938.
## Rationale for this change
The PR tier still ran Spark 4.1 (catalyst and `sql_core`) and Iceberg 1.11
on every push to every pull request. Review is now largely agent-driven, and so
is responding to review, so a pull request goes through several more
push-review-push rounds before it is queued, and there are more contributors
with PRs in flight. Each of those pushes paid for the whole Spark 4.1 build and
the Iceberg 1.11 run, and the merge queue runs both against the merge result
before anything lands anyway.
A sample of the last 100 `pull_request` runs of `ci.yml` (about 34 hours,
2026-09-13 to 2026-09-15):
| Suite | Ran in | Runner-minutes | Share of PR total | Sole
failing suite |
| --------------------- | ------ | -------------- | ----------------- |
------------------ |
| Spark SQL (4.1, core) | 36 | 5,679 | 16% | 0
runs |
| Iceberg 1.11 | 35 | 5,736 | 17% | 0
runs |
| Linux build | 42 | 20,197 | 58% | 1
run |
| all jobs | | 34,597 | |
|
That is about 70 PR runs a day and about 135 runner-hours a day on the two
suites, and in this window neither one was the job that turned a PR run red.
The trade-off is that a Spark SQL or Iceberg regression now surfaces in the
queue rather than on the PR, where a red run evicts the entry. Reviewers should
apply `run-spark-4.1-tests` or `run-iceberg-tests` before queuing anything that
touches the serde, the planner, a native operator, or the Iceberg paths.
Continues #5843, #5871 and #5930.
## What changes are included in this PR?
- `dev/ci/compute-changes.py`: `spark_4_1` and `iceberg_1_11` move from
`["pr", "queue"]` to `["queue", "label:..."]`. `run-spark-4.1-tests` gates both
Spark 4.1 outputs so it runs every module in the one 4.1 build;
`run-spark-4.1-hive-tests` still selects only the hive shards.
`run-iceberg-tests` now opts a PR into all four Iceberg versions.
- `dev/ci/check-ci-config.py`: `PR_TIER` shrinks to the Linux build; new
`synchronize` and `labeled` cases pin the `run-spark-4.1-tests` routing,
including the hive label applied on top of the suite label.
- `ci.yml`, `.github/workflows/README.md`, `dev/ci/spark-sql-modules.py`:
comments, tier lists, diagram and "what runs when" table. The README used to
say `run-spark-4.1-tests` gates nothing; that label did not exist at all. It is
created now, with the same description style as the other queue labels.
- `docs/source/contributor-guide/ci.md`, `spark-sql-tests.md`,
`iceberg-spark-tests.md`: tier table, label table, and when to apply a label.
## How are these changes tested?
`python3 dev/ci/check-ci-config.py` passes with the new `POLICY_CASES`,
`actionlint` passes on `ci.yml`, and `prettier --check` passes on the edited
Markdown. The `run-spark-4.1-tests` label exists on the repository. This PR's
own run is the first exercise of the new PR tier: it touches `dev/ci/**` and
`ci.yml`, so the Linux build runs and nothing else should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
