This is an automated email from the ASF dual-hosted git repository.
amoghrajesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 2cf2a1af91c Explicitly state 'durable execution' in airflow docs
(#71863)
2cf2a1af91c is described below
commit 2cf2a1af91cd86c19ed724f87af014dbd60fdb05
Author: Amogh Desai <[email protected]>
AuthorDate: Wed Aug 26 10:23:33 2026 +0530
Explicitly state 'durable execution' in airflow docs (#71863)
---
.../docs/core-concepts/resumable-tasks.rst | 23 ++++++++++++++++++++--
.../docs/core-concepts/task-state-store.rst | 3 +++
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/airflow-core/docs/core-concepts/resumable-tasks.rst
b/airflow-core/docs/core-concepts/resumable-tasks.rst
index 98df3d45e60..163e11b3346 100644
--- a/airflow-core/docs/core-concepts/resumable-tasks.rst
+++ b/airflow-core/docs/core-concepts/resumable-tasks.rst
@@ -32,8 +32,27 @@ entire polling duration, and if the worker process is
restarted or the host is p
retries from scratch, losing all the progress made. Depending on the operator,
that means the external
job is submitted again, creating a duplicate run in context of the external
system.
-Airflow recommends three approaches for handling long-running external work.
Understanding the trade-offs
-between them helps you choose the right one for your situation.
+.. _concepts-durable-execution:
+
+Durable execution
+-----------------
+
+Surviving that failure mode is what **durable execution** means: a task
outlives the loss of the
+process running it and can continue / re-attach from where it stopped, rather
than repeating work
+already done or submitting a duplicate job to an external system.
+
+The :doc:`task state store <task-state-store>` is the mechanism Airflow
provides to achieve it. It
+is the only per-task-instance storage that outlives a worker crash and is
still readable by the
+next attempt, which is what lets a retry recover a checkpoint or an external
job identifier written
+by the attempt before it. Durable execution is the outcome; the task state
store is how you get it.
+
+Operator documentation across the provider ecosystem uses the same term for
the operator-level
+feature built on this mechanism, normally exposed as a ``durable`` parameter.
Spark, Kubernetes,
+Databricks, Snowflake, BigQuery, Redshift and Glue operators each have a
"Durable execution"
+section describing what they persist and how they reconnect on retry.
+
+Airflow recommends three approaches for achieving this with long-running
external work.
+Understanding the trade-offs between them helps you choose the right one for
your situation.
.. _concepts-resumable-tasks-deferrable:
diff --git a/airflow-core/docs/core-concepts/task-state-store.rst
b/airflow-core/docs/core-concepts/task-state-store.rst
index cd5ffbc8b5d..c4f7dfa6e40 100644
--- a/airflow-core/docs/core-concepts/task-state-store.rst
+++ b/airflow-core/docs/core-concepts/task-state-store.rst
@@ -32,6 +32,9 @@ Task State Store
Task store is a persistent key/value store scoped to a single task instance
(``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker
crashes and task retries within the same Dag run, making it suitable for
storing external job IDs, intra-task checkpoints, and progress metadata.
+Because it outlives a worker crash and stays readable by the next attempt, the
task state store is the mechanism behind :ref:`durable execution
<concepts-durable-execution>`, where a task continues
+from where it stopped instead of repeating work or submitting a duplicate
external job. Provider operators that advertise a ``durable`` parameter are
built on the API described here.
+
Data persisted via task state store is accessed through the task context via
``context["task_state_store"]`` and exposes the synchronous methods ``get``,
``set``, ``delete``, and ``clear``, plus the async counterparts ``aget``,
``aset``, ``adelete``, and ``aclear`` for use inside ``async`` tasks.
