potiuk opened a new pull request, #67506: URL: https://github.com/apache/airflow/pull/67506
`Client.__init__` did not set explicit `httpx.Limits`, so each Client inherited httpx's defaults of `max_connections=100` and `max_keepalive_connections=10`. The supervisor's own [`_ensure_client`](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1119) capped at `max_connections=10, max_keepalive_connections=1` — so the two code paths that build a task-sdk Client diverged by an order of magnitude in their pool behaviour, with no documented reason and surprising resource use under high concurrency. Reported as F-011 in the [`apache/tooling-agents` L3 task-sdk sweep `0920c77`](https://github.com/apache/tooling-agents/issues/24). ## Change Standardise on a single bounded default at the Client level via `kwargs.setdefault("limits", httpx.Limits(max_keepalive_connections=5, max_connections=20))`. The supervisor's lower-cap path is unchanged (it sets `limits` explicitly before passing to the Client), and any caller that needs different limits can still pass `limits=...` via kwargs. Choosing `(5, 20)` rather than `(1, 10)`: a non-supervisor Client (e.g. instantiated directly by tests, CLI, or future callers) may handle higher per-instance concurrency than a single task subprocess, so the default sits between the supervisor's tight cap and httpx's untenably-high default. ## Test plan - [x] `test_default_pool_limits_are_bounded` — instantiates a real (non-mock-transport) Client and asserts `pool._max_connections == 20` and `pool._max_keepalive_connections == 5`. - [x] `test_pool_limits_can_be_overridden` — `limits=httpx.Limits(...)` kwarg overrides the default. - [x] `prek run ruff` clean. - [x] `prek run mypy-task-sdk` clean. - [x] Full `test_client.py` suite: 117 passed. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.7) Generated-by: Claude Code (Opus 4.7) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
