andygrove opened a new pull request, #6000: URL: https://github.com/apache/datafusion-comet/pull/6000
## Which issue does this PR close? Relates to #5999. It does not close it: that issue is about fixing the Miri job, this documents how to notice a job in that state in the first place. - [Rendered page — CI guide](https://github.com/andygrove/datafusion-comet/blob/docs-scheduled-ci-health/docs/source/contributor-guide/ci.md#checking-that-the-scheduled-runs-are-healthy) - [Rendered page — release process](https://github.com/andygrove/datafusion-comet/blob/docs-scheduled-ci-health/docs/source/contributor-guide/release_process.md#release-preparation) ## Rationale for this change After #5939 and #5963, most of our coverage of the non-default Spark and Iceberg versions runs in the nightly tier rather than on pull requests. That was the right trade for cost, but it moves a lot of signal onto a schedule, and a scheduled run has no pull request to turn red. `ci.yml` has a `nightly_report` job that opens a `ci-nightly-failure` issue, so that tier is covered. `miri.yml` and `publish_snapshot.yml` have no such step — and both are currently broken without anyone having noticed. Miri has failed 79 nights running (#5999) and the nightly SNAPSHOT publish has never succeeded since it landed in #5902, failing with a 401 against repository.apache.org. There is also a subtler failure I could not find documented anywhere. For `ci.yml`, a green `Required Checks` does not mean the nightly suites ran. The path filters and the diff base are both allowed to select nothing, which is correct on a documentation-only day, but a run that tested nothing is indistinguishable from a run that tested everything if you only look at the verdict. If the diff base ever drifts, the nightly goes green forever while testing nothing. I checked the nightly tier itself and it is working — the 2026-09-16 and 2026-09-17 runs each completed about 40 Spark SQL and Iceberg suite jobs, with shards running 25 to 67 minutes, and the second run correctly resolved its diff base to the first run's head. So this PR is documentation only; there is nothing to fix in the tier. ## What changes are included in this PR? A new "Checking that the scheduled runs are healthy" section in the CI contributor guide, listing the four schedules and which of them report their own failures, with a `gh api` loop over the three daily ones and guidance on reading the output for the two distinct failures — a schedule that stopped firing versus a streak of red nights. It also gives a second command that counts the nightly's suite jobs by conclusion, so "green because everything passed" can be told apart from "green because nothing ran." The release process gets a checklist item and a paragraph under Release Preparation pointing at that section, so this gets looked at once per release rather than only when someone happens to wonder. The documented commands use `gh api` rather than `gh run list --event=schedule`. The latter is not reliable: while investigating this it omitted that morning's `ci.yml` scheduled run entirely, and on another call returned three-week-old `miri.yml` runs as the latest three. ## How are these changes tested? Documentation only, so no tests. Every command in the new section was run against the live repository and produced the output the surrounding text describes. `prettier --check` and `python3 dev/ci/check-ci-config.py` both pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
