steveahnahn opened a new pull request, #69631: URL: https://github.com/apache/airflow/pull/69631
### Problem A deferred `RedshiftDataOperator` parks its query in the triggerer, so the operator's own `on_kill` no longer runs once the task is deferred. When a user marks that task failed, clears it, or marks it success, `RedshiftDataTrigger` had no `on_kill` hook, so the Redshift statement kept running against the cluster or workgroup even though the operator already cancels the statement on kill in the non-deferred path. On Redshift Serverless a runaway statement keeps billing RPU hours, and on a provisioned cluster it holds a WLM slot, until it finishes on its own. ### Change `RedshiftDataTrigger` gains an `on_kill` that cancels the running statement when the user acts on the deferred task, using the hook's async connection (`cancel_statement`). This matches the behaviour already shipped for the EMR, Dataproc, BigQuery, and Dataflow triggers, and closes the last common AWS gap from the deferrable `on_kill` work tracked in #36090. A `cancel_on_kill` flag (default `True`) is added to both `RedshiftDataOperator` and `RedshiftDataTrigger` so users can opt out, and the operator threads it into the trigger at the defer site. Cancellation is best effort: failures while cancelling are logged and swallowed so the trigger teardown is not blocked. ### Precedent * #65740 `EmrServerlessStartJobTrigger` * #65742 `DataprocSubmitTrigger` / `DataprocSubmitJobDirectTrigger` * #66704 `BigQueryInsertJobTrigger` ### Tests Trigger tests cover cancel on kill, the disabled and missing statement id no ops, best effort error swallowing, and serialization of the new flag. Operator tests cover the `cancel_on_kill=False` guard and that the flag is threaded into the trigger on defer. Every new assertion fails without the source change. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5) Generated-by: Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
