Leondon9 opened a new pull request, #73719:
URL: https://github.com/apache/airflow/pull/73719
On a deferrable EMR Serverless job **failure or timeout**, the real failure
reason was masked by a `KeyError`.
## The bug
`EmrServerlessStartJobTrigger` emits a success event carrying `job_details`
(`{application_id, job_id}`), but a **failure** event carrying only `{status,
message}`. `EmrServerlessStartJobOperator.execute_complete` then read
`event["job_details"]["application_id"]` to cancel the run before raising — so
on any real failure it raised `KeyError: 'job_details'`, hiding the actual
reason. On resume `execute()` has not run, so `self.job_id` is also unset,
meaning the cancel had no run id to act on even if `job_details` had been
present.
Found while validating #73701 on live AWS: a genuine job failure surfaced
only as `KeyError: 'job_details'` in the task logs, not the underlying Spark
error.
## The fix
- The trigger now includes `job_details` in the **failure** event too,
consistent with the success event.
- `execute_complete` reads the run id from the event (falling back to
`self`) so it works on resume; cancels the run defensively (guarded, so an
already-terminal run cannot mask the failure); and raises the **actual**
failure message instead of a generic string.
This is an independent, pre-existing bug on `main`, unrelated to #73701 /
#73702 — it was just noticed while validating them.
## Testing
- New tests: the trigger failure event carries `job_details`;
`execute_complete` surfaces the real message; a failure event without
`job_details` no longer raises `KeyError`; a cancel error does not mask the
real reason.
- Full `test_emr_serverless.py` + EMR trigger suites green (92 passed).
- Provider `mypy` and other Docker-dependent prek hooks were not run locally
(Docker unavailable); CI will validate them.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
