andygrove commented on code in PR #6000: URL: https://github.com/apache/datafusion-comet/pull/6000#discussion_r4041884876
########## docs/source/contributor-guide/ci.md: ########## @@ -244,6 +244,58 @@ stays in one place. When you pick up a nightly failure: Dispatching `ci.yml` from the Actions page with **Run workflow** runs every tier, including the nightly suites, if you need a result before the next scheduled run. +## Checking that the scheduled runs are healthy + +A scheduled run has no pull request to turn red, so when one breaks, nothing puts it in front of +anyone. Comet has three daily schedules plus a weekly one, and two of the daily ones report nothing +when they fail: + +| Workflow | Cron (UTC) | Reports a failure? | +| ---------------------- | ------------ | ------------------------------------- | +| `publish_snapshot.yml` | `0 3 * * *` | no | +| `miri.yml` | `0 4 * * *` | no | +| `ci.yml` nightly tier | `0 6 * * *` | yes, a `ci-nightly-failure` issue | +| `codeql.yml` | `16 4 * * 1` | yes, to the repository's Security tab | + +So the two without a reporting step have to be looked at deliberately. Check all of them at once: + +```sh +for wf in ci.yml miri.yml publish_snapshot.yml; do + gh api "repos/apache/datafusion-comet/actions/workflows/$wf/runs?event=schedule&per_page=5" \ + --jq ".workflow_runs[] | \"$wf\t\(.created_at[0:10])\t\(.conclusion)\t\(.html_url)\"" +done +``` + +Read the output for two different things: + +- **A missing row.** If the most recent run is not from the last day or two, the schedule itself + stopped firing. GitHub drops scheduled runs under load and disables them entirely in a repository + with no activity for 60 days, and it does not announce either. Confirm the workflow is still + enabled with `gh api repos/apache/datafusion-comet/actions/workflows --jq '.workflows[] | "\(.state)\t\(.path)"'`, + and re-enable it from the Actions page if it is `disabled_inactivity`. +- **A run of failures.** One red night is a flake or a real regression, and for `ci.yml` there is an + issue open about it. Several consecutive red nights on `miri.yml` or `publish_snapshot.yml` means + nobody has looked; treat the streak, not the newest run, as the thing to explain. + +For the `ci.yml` nightly, green on its own does not mean the suites ran. Path filters and the diff +base are both allowed to select nothing — a documentation-only day legitimately runs no suite at +all — and a run that tested nothing reports exactly the same green `Required Checks` as a run that +tested everything. Confirm the suites actually ran: + +```sh +run=$(gh api "repos/apache/datafusion-comet/actions/workflows/ci.yml/runs?event=schedule&per_page=1" \ + --jq '.workflow_runs[0].id') +gh run view "$run" --json jobs \ Review Comment: Good catch, thanks — fixed in bacd419. You are right that the two halves of that snippet disagree with each other: the run ID is pinned to upstream and the lookup that consumes it is not, so it only works if your checkout happens to have upstream selected. I reproduced the 404 the same way, and with `--repo apache/datafusion-comet` added it returns the 40 successes and five skips the surrounding text describes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
