potiuk opened a new pull request, #71550: URL: https://github.com/apache/airflow/pull/71550
Checking whether a constraints file is up to date asks PyPI about every pinned package, one blocking request after another. `constraints-source-providers-3.10.txt` pins 768 packages, and at ~3.5s per request on a runner that sweep is the entire runtime of the `Deps <py>:constraints-source-providers` jobs — comparing the 2026-08-07 and 2026-08-13 canaries, the same 768 packages took 42.4 min and 47.1 min, so the job's duration is package count times PyPI latency and it grows with every dependency added. The requests are independent, so they now run on a small pool. Notes on the shape of it: * Batched rather than prefetched: a single project's release history can be megabytes (`/pypi/botocore/json` is ~3.7MB), so only one batch is ever held in memory. * Results are consumed in constraints-file order, so the printed table is unchanged. * A failed fetch is carried as its exception and re-raised at the same point in the loop, keeping per-package error reporting identical. * Each request also gets a timeout — a stalled connection previously hung the job until CI killed it. The request count itself is inherent to the check: it asks "is this pin behind its latest release, by how many versions, and how stale is it", which needs one PyPI lookup per pin. `--diff-mode` filters after the fetch, so the narrower modes cost the same. The PEP 691/700 simple JSON API was measured as an alternative and is only ~1.5x smaller than the full JSON, so it is not worth switching to. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
