potiuk commented on PR #71550: URL: https://github.com/apache/airflow/pull/71550#issuecomment-5341811421
Good question. PyPI is served behind a CDN and doesn't publish a hard limit for the JSON API, so ten in flight is generally absorbed at the edge — but a burst can still come back 429, so the concern is real. I've added bounded retries for exactly that: 429 and 5xx are retried, waiting as long as PyPI asks via `Retry-After` when it sends one and backing off exponentially otherwise. A 404 is deliberately not retried, since that's an answer rather than congestion. I kept the parallelism at 10 rather than scaling it down adaptively — worth revisiting if we ever see throttling in practice, but that adds a moving part for a problem we haven't observed yet. --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
