kaxil opened a new pull request, #66186:
URL: https://github.com/apache/airflow/pull/66186
## Summary
Each registry build does ~2N sequential `urllib.request.urlopen` calls to
pypistats.org and pypi.org -- one per provider per endpoint, with 5-10s
timeouts each. With ~86 providers that's ~172 serialised HTTP calls, which
dominated wall-clock (recent successful full builds: 26-30 min, much of it
spent in those fetches). A single flaky response zeroes that provider's
downloads, but the build still serialises on the next call regardless.
## Fix
Batch the calls through `concurrent.futures.ThreadPoolExecutor` with
`max_workers=16`. New helper `fetch_pypi_data_parallel(package_names)`
pre-fetches everything once and returns a `{package_name: (downloads_dict,
dates_dict)}` map; the per-provider loop in `main()` is then a dict lookup.
The pattern matches the existing `ThreadPoolExecutor` use in
[`extract_parameters.py`](https://github.com/apache/airflow/blob/main/dev/registry/extract_parameters.py)
(max_workers=8 for inventory fetches) and
[`extract_versions.py`](https://github.com/apache/airflow/blob/main/dev/registry/extract_versions.py).
## Why max_workers=16
- pypistats.org and pypi.org each tolerate 16 parallel connections without
rate-limiting in practice.
- The two existing parallelised paths use 8 (sphinx inventory fetches) --
but those download larger files. PyPI calls are simpler/faster, so a higher
worker count gives bigger throughput.
- 86 providers / 16 workers ≈ 6 batches; well under the per-build budget
even with retries.
## Per-package failure isolation
`fetch_pypi_downloads` and `fetch_pypi_dates` already return zero-value
defaults on any error (network timeout, 404, decode error), so a single bad
response only zeroes that one package -- it cannot poison the dict for any
other package or cause `as_completed` to raise. New test
`test_per_package_failure_does_not_abort_others` covers this.
## Verification
- 62 tests pass (`pytest dev/registry/tests/test_extract_metadata.py`),
including 3 new tests:
- `test_returns_one_entry_per_package`
- `test_empty_input`
- `test_per_package_failure_does_not_abort_others`
- `prek run --files dev/registry/extract_metadata.py
dev/registry/tests/test_extract_metadata.py`: all hooks pass (license header,
ruff, ruff-format, codespell, ...).
## Expected wall-clock impact
A full build's PyPI-fetch phase should drop from ~5-8 min (sequential,
dominated by per-call timeouts) to ~30-60 s (16 in parallel, bounded by the
slowest 5-10s timeout per batch). That brings full-build runtime comfortably
under the 30-minute job timeout and reduces the surface for transient pypistats
slowdowns to delay registry updates.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes -- Claude Code (Opus 4.7)
Generated-by: Claude Code (Opus 4.7) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
