This is an automated email from the ASF dual-hosted git repository. vatsrahul1001 pushed a commit to branch backport-2cf2a1af91-v3-3-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 426140317ddcda0043064cae4a423dca6ff43dc8 Author: Amogh Desai <[email protected]> AuthorDate: Wed Aug 26 10:23:33 2026 +0530 Explicitly state 'durable execution' in airflow docs (#71863) (cherry picked from commit 2cf2a1af91cd86c19ed724f87af014dbd60fdb05) --- .../docs/core-concepts/resumable-tasks.rst | 23 ++++++++++++++++++++-- .../docs/core-concepts/task-state-store.rst | 3 +++ 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/airflow-core/docs/core-concepts/resumable-tasks.rst b/airflow-core/docs/core-concepts/resumable-tasks.rst index ecb2bcaf4c0..45c993bda42 100644 --- a/airflow-core/docs/core-concepts/resumable-tasks.rst +++ b/airflow-core/docs/core-concepts/resumable-tasks.rst @@ -32,8 +32,27 @@ entire polling duration, and if the worker process is restarted or the host is p retries from scratch, losing all the progress made. Depending on the operator, that means the external job is submitted again, creating a duplicate run in context of the external system. -Airflow recommends three approaches for handling long-running external work. Understanding the trade-offs -between them helps you choose the right one for your situation. +.. _concepts-durable-execution: + +Durable execution +----------------- + +Surviving that failure mode is what **durable execution** means: a task outlives the loss of the +process running it and can continue / re-attach from where it stopped, rather than repeating work +already done or submitting a duplicate job to an external system. + +The :doc:`task state store <task-state-store>` is the mechanism Airflow provides to achieve it. It +is the only per-task-instance storage that outlives a worker crash and is still readable by the +next attempt, which is what lets a retry recover a checkpoint or an external job identifier written +by the attempt before it. Durable execution is the outcome; the task state store is how you get it. + +Operator documentation across the provider ecosystem uses the same term for the operator-level +feature built on this mechanism, normally exposed as a ``durable`` parameter. Spark, Kubernetes, +Databricks, Snowflake, BigQuery, Redshift and Glue operators each have a "Durable execution" +section describing what they persist and how they reconnect on retry. + +Airflow recommends three approaches for achieving this with long-running external work. +Understanding the trade-offs between them helps you choose the right one for your situation. .. _concepts-resumable-tasks-deferrable: diff --git a/airflow-core/docs/core-concepts/task-state-store.rst b/airflow-core/docs/core-concepts/task-state-store.rst index 7702fb90f68..6d64ac3fb36 100644 --- a/airflow-core/docs/core-concepts/task-state-store.rst +++ b/airflow-core/docs/core-concepts/task-state-store.rst @@ -30,6 +30,9 @@ Task State Store Task store is a persistent key/value store scoped to a single task instance (``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker crashes and task retries within the same Dag run, making it suitable for storing external job IDs, intra-task checkpoints, and progress metadata. +Because it outlives a worker crash and stays readable by the next attempt, the task state store is the mechanism behind :ref:`durable execution <concepts-durable-execution>`, where a task continues +from where it stopped instead of repeating work or submitting a duplicate external job. Provider operators that advertise a ``durable`` parameter are built on the API described here. + Data persisted via task state store is accessed through the task context via ``context["task_state_store"]`` and exposes four methods: ``get``, ``set``, ``delete``, and ``clear``.
