potiuk opened a new pull request, #68368: URL: https://github.com/apache/airflow/pull/68368
Implements the monitoring proposed on the dev list in the *"Rebase your PRs for faster CI times"* thread: an automated warning when CI run times on `main` drift above their recent trend, so slowdowns are caught early instead of at the timeout cliff. ### What it does A new daily scheduled workflow (`ci-duration-monitor.yml`) runs `scripts/ci/analyze_ci_job_durations.py`, which: - Fetches the recent **`schedule`-event runs of `ci-amd.yml` on `main`** — the post-merge canaries that are the only CI signal on `main` (PR runs are excluded so they can't pollute the trend). - Computes the **wall-clock duration of each run** and the **per-job duration** (success-only, so a cancelled/skipped job doesn't distort its own trend). - Compares the latest run against a **robust median baseline** of the preceding runs, flagging a regression only when the increase clears **both** a relative threshold (default +25%) **and** an absolute floor (5 min overall / 3 min per job) — so short, noisy jobs don't raise spurious alerts. - Posts to the `internal-airflow-ci-cd` Slack channel **only when a regression is detected** (and always writes a step-summary table); otherwise it stays quiet. All thresholds, the window size, the workflow/branch/event, and the channel are configurable via env vars. ### Testing - 27 unit tests in `scripts/tests/ci/test_analyze_ci_job_durations.py` covering timestamp/duration parsing, the median + dual-threshold regression logic (relative floor, absolute floor, outlier-robustness), event/short-run filtering, and Slack formatting. - Ran the script live against `apache/airflow`: the graceful "not enough data" path, the `event=schedule` filtering, and the per-job analysis all behave as expected. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
