kaxil opened a new pull request, #73527: URL: https://github.com/apache/airflow/pull/73527
The `Static checks` job spends about five minutes of every pull request run re-compressing and re-uploading a prek cache it has just downloaded. This PR removes that work on cache hits and shrinks the cache itself, for roughly a 6 to 7 minute saving on a job that currently takes 25 to 40 minutes. Timings from a typical run ([job 106662807685](https://github.com/apache/airflow/actions/runs/35700940143/job/106662807685), 25.3 min total): | Step | Time | |---|---| | Download prek cache stash (2.7 GB gzipped, 7.2 GB extracted) | 3.0 min | | Extract | 0.7 min | | `prek install-hooks` (cache hit) | 1 s | | Re-tar the same cache | 3.5 min | | Re-upload it | 1.8 min | **Save only when there is something new to save.** The stash restore looks for the PR branch first and then falls back to the base branch, so a PR on a cache hit is holding the `main` stash and would upload a byte-identical copy under its own branch name. The `install-prek` action now skips the tar and upload steps when the tarball was restored and the event is a `pull_request`. Cache misses still save, and non-PR runs (the canary and scheduled runs on `main`) still save on every run so the base-branch stash stays inside its two-day retention window. `save-cache: true` is only set by the two static-check jobs, and both are reached only from `ci-amd.yml` and `ci-arm.yml`, so `pull_request` is the only PR-shaped event that reaches this condition. **Drop the skill-eval environments from the cache.** `prek install-hooks` builds environments for every configured hook, including manual-stage ones that CI never runs. The three `skill-eval` hooks pull promptfoo plus the Anthropic and OpenAI agent SDKs into two node environments, which are about 3.5 GB (measured on a local prek cache; the CI log has no per-environment breakdown, only the 7.2 GB total). Skipping them in the CI install roughly halves what every job downloads and extracts. Because `--skip` only avoids building environments and does not remove restored ones, the cache key moves from `cache-prek-v9` to `cache-prek-v10`, so the first run after merge builds the cache from scratch without them instead of re-saving the restored oversized one. The miss path also compresses with `pigz` when it is on the runner (it ships in the `ubuntu-22.04` image) and falls back to `gzip` otherwise. The output is plain gzip, so the restore side is unchanged and the cache key does not need a bump. The remaining large blocks in this job are the CI image download and load (5.3 min) and the sequential hook run (9.5 min), which are separate changes. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
