andygrove opened a new issue, #5938: URL: https://github.com/apache/datafusion-comet/issues/5938
## What is the problem the feature request solves? The PR tier still runs Spark 4.1 (catalyst and `sql_core`) and Iceberg 1.11 on every push to every pull request. That made sense when a PR was pushed a handful of times before merging. Two things changed: - Pull request review is now largely agent-driven, and so is responding to review feedback. A PR goes through several more push-review-push rounds before it is queued, and each round re-runs both suites. - There are more contributors, so there are more PRs in flight at once. A sample of the last 100 `pull_request` runs of `ci.yml` (2026-09-13 20:35 UTC to 2026-09-15 06:15 UTC, about 34 hours): | Suite | Ran in | Runner-minutes | Share of PR total | Sole failing suite | | ------------------------------ | ------ | -------------- | ----------------- | ------------------ | | Spark SQL (4.1, core) | 36 | 5,679 | 16% | 0 runs | | Iceberg 1.11 | 35 | 5,736 | 17% | 0 runs | | Linux build | 42 | 20,197 | 58% | 1 run | | all jobs | | 34,597 | | | That is roughly 100 PR runs and 136 runner-hours a day spent on the two suites, and in this window neither one was the job that turned a PR run red. The merge queue runs both against the merge result before anything lands regardless. ## Describe the potential solution Move every Spark SQL and Iceberg suite behind the merge queue by default, leaving the Linux build as the whole PR tier: - `spark_4_1`: `["queue", "label:run-spark-4.1-tests"]` - `spark_4_1_hive`: also gated by `run-spark-4.1-tests`, so that label runs every module in the one 4.1 build while `run-spark-4.1-hive-tests` still selects only the hive shards - `iceberg_1_11`: `["queue", "label:run-iceberg-tests"]`, so one label opts a PR into all four Iceberg versions Create the `run-spark-4.1-tests` label (it did not exist, despite the workflows README saying it "gates nothing"), pin the new routing in `dev/ci/check-ci-config.py`, and update the workflows README and the contributor guide. The trade-off is that a Spark SQL or Iceberg regression now surfaces in the queue rather than on the PR, where a red run evicts the entry. Reviewers should apply the labels before queuing anything that touches the serde, planner, native operators, or the Iceberg paths. This continues #5870 and #5929. ## Additional context Follows the tiering in #5843, #5871 and #5930. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
