moomindani opened a new pull request, #72304: URL: https://github.com/apache/airflow/pull/72304
`DatabricksHook.get_run` and its async twin read only the first page of `2.2/jobs/runs/get`. The Jobs API returns at most 100 entries of a run's `tasks` and `job_clusters` per page and hands back a `next_page_token` for the rest, so on a job with more tasks than that the run came back carrying an arbitrary subset of them — the entries are not returned in declaration order, so which ones went missing was not predictable. The visible effect is in `extract_failed_task_errors[_async]`, which walks `run_info["tasks"]` to attach each failed task's own error to the message the task fails with: a failure on a later page falls back to the generic run state message instead. `get_run_tasks` already paginated, so it now delegates to `get_run` and the two cannot drift apart again. Verified against a live workspace, using a job of 101 `condition_task` entries (no compute) plus one failing notebook task: * before: `get_run()["tasks"]` returned 100 entries while `get_run_tasks()` returned 103, and the keys missing from the former were an arbitrary trio * after: both return the complete list, with the run-level fields unchanged * `job_clusters` is paginated too rather than repeated on each page — a job with two job clusters returns both on page 1 and none on page 2 — so merging it is correct Both new unit tests fail without the change (`call_count 1 == 2`), and the pre-existing `get_run_tasks` multi-page test passes unchanged. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
