amoghrajesh commented on code in PR #72100: URL: https://github.com/apache/airflow/pull/72100#discussion_r3869230261
########## airflow-core/newsfragments/72100.significant.rst: ########## @@ -0,0 +1,39 @@ +Clearing a task now discards its task state store entries + +Clearing a task instance discards its ``task_state_store`` entries, so the next attempt starts from +the beginning instead of resuming from a checkpoint or reconnecting to an external job recorded by +the attempt that was cleared. + +Retries are unaffected. They keep task state exactly as before, which is what crash recovery relies +on. Only a deliberate clear discards. + +**Why** + +Clearing means "run this again". A checkpoint records how far a task got, not what it got there +with, so resuming after the code or the upstream data changed left work done before the fix in place +and silently mixed it with the corrected work. Clearing a task whose external job had already +succeeded was worse: the operator read the stored result back and returned in seconds having run +nothing. + +**Keeping the old behaviour** + +Pass ``keep_task_state=True`` to the clear task instances endpoint, or tick "keep task state" in the +clear dialog. Use it when nothing about the inputs or the code changed and the task should carry on +where it stopped, or when an external job is still running and you want the next attempt to +reconnect rather than submit a duplicate. + +Operators with durable execution are worth particular attention. Clearing a *failed* task never runs +``on_kill``, so an external job that outlived its worker is still running, and discarding the stored Review Comment: Yep, it is and not specific to durable tasks. `on_kill()` only fires for SIGTERM, or `execution_timeout` being exceeded. A task that fails by raising never calls it. So a durable task that exhausts its retries can leave the external job running with nothing tracking it! Checked the current durable operators. Eight of nine put cancellation only in `on_kill` (Glue, Redshift Data, BigQuery, Snowflake, Livy, SparkSubmit, both Databricks ones), so none clean up on failure. KPO is the exception, and by a different route: its cleanup sits in a finally in `execute_sync` honouring `on_finish_action`, so it runs on any exit. ResumableJobMixin has no cleanup path of its own. I'd keep it out of this PR since it's terminal-failure cleanup rather than clear semantics, but if a terminally failed durable task cancelled its job, there'd be nothing left running for a later clear to duplicate, and most of this caveat goes away. I'll open an issue for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
