amoghrajesh opened a new issue, #71489: URL: https://github.com/apache/airflow/issues/71489
`GlueJobOperator._find_job_run_id_by_task_uuid` (the task-UUID scan fallback, used when neither `task_state_store` nor a cached XCom id has a prior run) has a few pre-existing quality issues, carried over verbatim from `main`, that are worth cleaning up: 1. **Unbounded pagination.** It's a `while True` over `get_job_runs(MaxResults=50)` with no page cap and no age cutoff. The no-match case — the prior attempt died before it ever called `StartJobRun`, which is the common retry shape — is exactly the one that walks the job's entire run history before giving up. 2. **Failure is silently swallowed.** The two `except Exception` blocks around this scan (and the XCom lookup before it) log at `warning` level and return `None`, after which the operator submits fresh — straight into `ConcurrentRunsExceededException` against a run that's still alive. There's no error-level signal telling anyone why. 3. **The `except Exception` is broader than it needs to be.** It should be narrowed to `except ClientError`, so a real bug in the surrounding code doesn't get silently absorbed the same way as a transient AWS-side issue. 4. **Missing IAM documentation.** Anyone relying on this fallback (via `resume_glue_job_on_retry=True` today, or explicit `durable=True` on Airflow <3.3 going forward) needs `glue:GetJobRuns` in their task policy, on top of `StartJobRun`/`GetJobRun`. The docs don't currently call that out. None of this is new — it's present on `main` today, gated behind the opt-in `resume_glue_job_on_retry` flag. Raised during review of #71211, initially flagged as more urgent because that PR was going to make the scan reachable by default; the default-flip was scoped back to explicit opt-in for Airflow <3.3 (and to a narrow crash-recovery window on 3.3+) in the same PR, so the original urgency no longer applies — but the underlying issues are still real for anyone who does hit this path. Fix direction: bound the walk (a page cap, or stop once a run's `StartedOn` predates the DAG run), narrow the exception handling, log the fallthrough at `error` level with the actual reason, and document the IAM requirement. See https://github.com/apache/airflow/pull/71211#discussion_r3764425788 for the original discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
