potiuk opened a new pull request, #75615: URL: https://github.com/apache/airflow/pull/75615
Jobs on the self-hosted RunsOn runners fail with exit code 137 when AWS reclaims their Spot instance. Nothing in the job log says why, so it looks like a broken test or an out-of-memory kill, although RunsOn re-runs the job by itself once the run finishes. Today's canary and #75612 were both hit. - `spot-interruption-notice.yml` checks the newest "Tests (AMD)" run of each PR every 10 minutes, including runs still in progress, and keeps one comment on the PR: which jobs were interrupted and that RunsOn re-runs them, then how the re-run ended. It runs on a schedule, so no PR run gets write access. - A canary's `notify-slack` no longer posts failures that are only Spot interruptions RunsOn will retry. It posts for RunsOn's re-run instead (which it previously never ran for, so recoveries went unreported), and still posts when RunsOn gives up after attempt 3. - An interruption is recognised only by RunsOn's annotation on a job that ran on a RunsOn runner, so real failures and annotations a PR's own workflow could add are never treated as one. - The API budget of the polling is measured and documented in `05_workflows.md`: about 2% of the repository's `GITHUB_TOKEN` limit. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Opus 5 Generated-by: Claude Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
