ColtenOuO opened a new pull request, #72388: URL: https://github.com/apache/airflow/pull/72388
### Sumarry In `execute_complete()`, the return value of `paginate_async()` was discarded, and the code called `process_response()` a second time with just the current page's `response`. `paginate_async()` already calls `process_response()` itself with the full accumulated `all_responses` list once pagination ends, so its result was thrown away and immediately overwritten with a single-page result. The practical effect: in synchronous mode, `HttpOperator` with a `pagination_function` returns a list of every page's response. In deferrable mode, it silently returned only the last page. Any Dag that switched a paginated `HttpOperator` to `deferrable=True` would quietly lose all pages but the last one, with no error raised. The existing regression test, `test_async_pagination`, could not catch this because both `execute_complete()` calls were wrapped in a single `contextlib.suppress(TaskDeferred)` block. The first call raises `TaskDeferred` (correctly, to request the next page), which is suppressed — but that also exits the `with` block immediately, so the second `execute_complete()` call and its `assert result == [...]` never executed. The test always reported as passing regardless of what the code actually returned. ### Changes - `paginate_async()` now owns the full response-building/return logic (including the non-paginated case, which previously lived in `execute_complete()`), and `execute_complete()` simply returns its result. - Rewrote `test_async_pagination` to use `pytest.raises(TaskDeferred)` around each deferring call individually, and assert on the final, non-raising resume call's result — so the test actually exercises the code path instead of skipping it. - Added `test_async_pagination_with_response_filter` to cover a `response_filter` receiving the full accumulated page list across three pages (two deferrals, one final page). - Added `test_async_execute_complete_without_pagination_returns_single_response` to lock in the unpaginated resume behavior, which is unchanged but now goes through the refactored `paginate_async()`. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Sonnet 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
