Thanks Amogh for adding the `on_kill` method to `BaseTrigger`! Without your change, the only way for me to achieve an explicit kill from the CLI or UI is by adding a `@teardown` task at the end of the external job submission for the "sentinel" purpose. Your change even makes this better, we can achieve the explicit kill with an atomic task!
Best, Jason On Fri, Apr 24, 2026 at 4:14 AM Jens Scheffler <[email protected]> wrote: > Thanks Amogh for adding this! > > In deed this was missing for a long time! > > You did not tell about but we can assume plan is to release this with > Airflow 3.3.0? > > On 23.04.26 18:03, Aritra Basu wrote: > > This is quite useful! Nice! > > Thanks and Regards, > > Aritra Basu > > > > > > On Thu, Apr 23, 2026 at 9:17 PM Amogh Desai <[email protected]> > wrote: > > > >> Hi Again, > >> > >> Created an issue tracking the migration of provider triggers from the > old > >> CancelledError > >> pattern to the new on_kill() hook: > >> https://github.com/apache/airflow/issues/65733 > >> > >> Three triggers need updating — Dataproc, EMR Serverless, and Kubernetes. > >> Each is a > >> contained change in a single file. Good first contribution if you want > to > >> get familiar with the > >> triggerer. > >> > >> Drop a comment on the issue if you want to pick one up. > >> > >> Thanks & Regards, > >> Amogh Desai > >> > >> > >> On Thu, Apr 23, 2026 at 7:40 PM Kaxil Naik <[email protected]> wrote: > >> > >>> Finally, this has been troubling users since Triggerer was added. > >>> > >>> On Thu, 23 Apr 2026 at 14:59, Elad Kalif <[email protected]> wrote: > >>> > >>>> This is great! well done! > >>>> > >>>> On Thu, Apr 23, 2026 at 4:42 PM Amogh Desai <[email protected]> > >>> wrote: > >>>>> Hey All, > >>>>> > >>>>> Wanted to call out a new hook we added to BaseTrigger — `on_kill()`, > >>> PR: > >>>>> https://github.com/apache/airflow/pull/65590. > >>>>> > >>>>> *Why does it exist?* > >>>>> > >>>>> If your trigger polls an external job (BigQuery, Databricks, > >> Dataproc, > >>>>> etc.), there was no clean way to cancel that job when > >>>>> a user kills the deferred task, either by marking it as success, > >>> failure, > >>>>> or clearing it. `cleanup()` wasn't the right place — > >>>>> it fires on every trigger exit including triggerer restarts and > >> rolling > >>>>> deploys, so putting cancellation there would kill in-flight > >>>>> work on every redeploy. > >>>>> > >>>>> `on_kill()` solves this. It only fires when a user explicitly acts on > >>> the > >>>>> task. Not on restart, not on redistribution. > >>>>> > >>>>> *How it works* > >>>>> > >>>>> The triggerer passes a private sentinel via > >> `asyncio.Task.cancel(msg)` > >>> to > >>>>> distinguish user kills from other exits — no DB > >>>>> round-trip needed. There's a configurable timeout (*[triggerer] > >>>>> on_kill_timeout, default 30s*) so a slow external API call > >>>>> cannot block the triggerer. > >>>>> > >>>>> Databricks is updated as a reference: > >>>>> https://github.com/apache/airflow/pull/65672 > >>>>> > >>>>> Both `DatabricksExecutionTrigger` and > >>>>> `DatabricksSQLStatementExecutionTrigger` now implement it. > >>>>> > >>>>> *Who benefits from this?* > >>>>> > >>>>> If you are someone building or maintaining a trigger that manages > >>>> external > >>>>> work, you can consider adopting to this hook > >>>>> and implementing it for your trigger class, Airflow 3.3 will launch > >>> this. > >>>>> No compat guard is needed, Airflow < 3.2 the method just never gets > >>>> called. > >>>>> *Next steps* > >>>>> > >>>>> I pulled some data from the repo and these are candidates for this > >>>>> migration. > >>>>> > >>>>> Google Provider: DataprocSubmitTrigger and > >>> DataprocSubmitJobDirectTrigger > >>>>> in the google provider. Currently catches > >>>>> CancelledError in run() and calls hook.cancel_job(job_id) after > >>> checking > >>>>> safe_to_cancel(). > >>>>> > >>>>> Amazon Provider: EmrServerlessStartJobTrigger. Same pattern. > >>>> CancelledError > >>>>> handler with safe_to_cancel() calling > >>>>> hook.conn.cancel_job_run(). Straightforward migration. > >>>>> > >>>>> CNCF provider: KubernetesPodTrigger in cncf/kubernetes. Has an > >>>>> on_kill_action parameter controlling whether to > >>>>> delete/patch the pod. The cancellation logic lives in a > >> CancelledError > >>>>> handler. Move it to on_kill(). > >>>>> > >>>>> Happy to help with reviews for these. > >>>>> > >>>>> > >>>>> Thanks & Regards, > >>>>> Amogh Desai > >>>>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
