bramhanandlingala opened a new pull request, #71574:
URL: https://github.com/apache/airflow/pull/71574
Closes #71489
## Summary
Bounds and hardens the Glue task-UUID scan fallback used on retry.
## Root Cause
The scan looped over `GetJobRuns` with no page cap. A no-match run —
the common retry case — walked the job's entire run history before
giving up. Failures were also swallowed by a broad `except Exception`
and logged only at `warning`, so a fallback failure gave no clear
signal before the operator submitted a fresh run into a still-active
one.
## Fix
- Added a page cap (`TASK_UUID_SCAN_MAX_PAGES`) and an early exit once
a run's `StartedOn` predates the DAG run (runs come back newest-first).
- Narrowed `except Exception` to `except ClientError`.
- Raised the fallthrough log level to `error` with the actual reason.
- Documented the `glue:GetJobRuns` IAM requiremen
Was generative AI tooling used ?
- [X] Yes - Claude
Generated-by: Claude following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
