This is an automated email from the ASF dual-hosted git repository.
vatsrahul1001 pushed a commit to branch v3-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-3-test by this push:
new 14261e91327 Explicitly state 'durable execution' in airflow docs
(#71863) (#73012)
14261e91327 is described below
commit 14261e9132747853c27f551382fe228bad276861
Author: Rahul Vats <[email protected]>
AuthorDate: Sat Sep 12 04:50:42 2026 +0530
Explicitly state 'durable execution' in airflow docs (#71863) (#73012)
(cherry picked from commit 2cf2a1af91cd86c19ed724f87af014dbd60fdb05)
Co-authored-by: Amogh Desai <[email protected]>
---
.../docs/core-concepts/resumable-tasks.rst | 23 ++++++++++++++++++++--
.../docs/core-concepts/task-state-store.rst | 3 +++
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/airflow-core/docs/core-concepts/resumable-tasks.rst
b/airflow-core/docs/core-concepts/resumable-tasks.rst
index ecb2bcaf4c0..45c993bda42 100644
--- a/airflow-core/docs/core-concepts/resumable-tasks.rst
+++ b/airflow-core/docs/core-concepts/resumable-tasks.rst
@@ -32,8 +32,27 @@ entire polling duration, and if the worker process is
restarted or the host is p
retries from scratch, losing all the progress made. Depending on the operator,
that means the external
job is submitted again, creating a duplicate run in context of the external
system.
-Airflow recommends three approaches for handling long-running external work.
Understanding the trade-offs
-between them helps you choose the right one for your situation.
+.. _concepts-durable-execution:
+
+Durable execution
+-----------------
+
+Surviving that failure mode is what **durable execution** means: a task
outlives the loss of the
+process running it and can continue / re-attach from where it stopped, rather
than repeating work
+already done or submitting a duplicate job to an external system.
+
+The :doc:`task state store <task-state-store>` is the mechanism Airflow
provides to achieve it. It
+is the only per-task-instance storage that outlives a worker crash and is
still readable by the
+next attempt, which is what lets a retry recover a checkpoint or an external
job identifier written
+by the attempt before it. Durable execution is the outcome; the task state
store is how you get it.
+
+Operator documentation across the provider ecosystem uses the same term for
the operator-level
+feature built on this mechanism, normally exposed as a ``durable`` parameter.
Spark, Kubernetes,
+Databricks, Snowflake, BigQuery, Redshift and Glue operators each have a
"Durable execution"
+section describing what they persist and how they reconnect on retry.
+
+Airflow recommends three approaches for achieving this with long-running
external work.
+Understanding the trade-offs between them helps you choose the right one for
your situation.
.. _concepts-resumable-tasks-deferrable:
diff --git a/airflow-core/docs/core-concepts/task-state-store.rst
b/airflow-core/docs/core-concepts/task-state-store.rst
index 7702fb90f68..6d64ac3fb36 100644
--- a/airflow-core/docs/core-concepts/task-state-store.rst
+++ b/airflow-core/docs/core-concepts/task-state-store.rst
@@ -30,6 +30,9 @@ Task State Store
Task store is a persistent key/value store scoped to a single task instance
(``dag_id`` + ``run_id`` + ``task_id`` + ``map_index``). It survives worker
crashes and task retries within the same Dag run, making it suitable for
storing external job IDs, intra-task checkpoints, and progress metadata.
+Because it outlives a worker crash and stays readable by the next attempt, the
task state store is the mechanism behind :ref:`durable execution
<concepts-durable-execution>`, where a task continues
+from where it stopped instead of repeating work or submitting a duplicate
external job. Provider operators that advertise a ``durable`` parameter are
built on the API described here.
+
Data persisted via task state store is accessed through the task context via
``context["task_state_store"]`` and exposes four methods: ``get``, ``set``,
``delete``, and ``clear``.
