This is an automated email from the ASF dual-hosted git repository.
amoghrajesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new c82b0f3bc7a Update provider docs for task state discard on clear
(#76968)
c82b0f3bc7a is described below
commit c82b0f3bc7a27d1e40548229de3f234b665a2f76
Author: Amogh Desai <[email protected]>
AuthorDate: Thu Nov 19 17:27:28 2026 +0530
Update provider docs for task state discard on clear (#76968)
---
providers/amazon/docs/operators/glue.rst | 41 +++++++++++++++---
.../docs/operators/redshift/redshift_data.rst | 49 ++++++++++++++++++----
providers/apache/spark/docs/operators.rst | 41 ++++++++++++++----
providers/common/ai/docs/durable_execution.rst | 19 +++++----
.../common/ai/docs/frameworks/openai_agents.rst | 7 ++--
providers/common/ai/docs/frameworks/strands.rst | 7 ++--
providers/databricks/docs/operators/run_now.rst | 48 +++++++++++++++++----
providers/databricks/docs/operators/submit_run.rst | 47 +++++++++++++++++----
providers/huggingface/docs/operators/datasets.rst | 38 +++++++++--------
providers/snowflake/docs/operators/snowflake.rst | 49 ++++++++++++++++++----
10 files changed, 262 insertions(+), 84 deletions(-)
diff --git a/providers/amazon/docs/operators/glue.rst
b/providers/amazon/docs/operators/glue.rst
index 6a4459b4898..d442c430af7 100644
--- a/providers/amazon/docs/operators/glue.rst
+++ b/providers/amazon/docs/operators/glue.rst
@@ -203,11 +203,42 @@ id won't be there for the next retry, and the operator
falls back to the XCom/sc
above rather than reconnecting via task state store. Avoid running cleanup on
a schedule shorter
than your longest ``retry_delay``.
-Clearing a task is treated the same as a retry, which matters specifically for
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes the stored run
+id by default from Airflow 3.4. Every other clear, such as ``airflow tasks
clear`` or clearing a
+whole Dag run, does so from Airflow 3.5. To keep it, tick **Keep task state
and resume** in the
+Clear dialog, or send ``"keep_task_state": true`` in the request. The **Keep
task state on clear**
+UI setting ticks that box by default. From Airflow 3.5, ``airflow tasks clear
--keep-task-state``
+keeps it too. The next attempt then reads the stored run id back. In
synchronous mode, what the next
+attempt does depends on the state of the Glue job run at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Job run state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running
+ - Fails with ``ConcurrentRunsExceededException`` until the first run
ends, with
+ ``concurrent_run_limit=1`` (the default). With a higher limit, submits
a new job run and
+ the first one keeps running.
+ - Reconnects to the running job run.
+ * - Succeeded
+ - Submits a new job run.
+ - Returns right away without starting another one.
+ * - Failed or stopped
+ - Submits a new job run.
+ - Submits a new job run.
+
+With ``deferrable=True``, ``durable=True`` and ``stop_job_run_on_kill=False``
(the default), the
+next attempt reconnects to a still-running job run on clear, whether or not
task state was kept.
+With ``stop_job_run_on_kill=True`` instead, it stops the previous run before
submitting a new one.
+
+With ``[state_store] clear_on_success`` on, the stored run id is deleted when
the task succeeds, so
+clearing a succeeded task submits again even when task state is kept.
+
+Keep task state when a job run is still running and nothing about the code or
inputs changed, so you
+don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` for
the ways of clearing.
To opt out and always start a fresh run on retry, set ``durable=False``:
diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst
b/providers/amazon/docs/operators/redshift/redshift_data.rst
index f4e100e3bdd..7f4a719ba92 100644
--- a/providers/amazon/docs/operators/redshift/redshift_data.rst
+++ b/providers/amazon/docs/operators/redshift/redshift_data.rst
@@ -99,15 +99,46 @@ the statement id won't be there for the next retry, and the
operator will submit
instead of reconnecting. Avoid running cleanup on a schedule shorter than your
longest
``retry_delay``.
-Clearing a task is treated the same as a retry, which matters specifically for
a task whose
-statement already succeeded: clearing does not delete the stored statement id,
so the next attempt
-reads it back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
-polling synchronously can cancel the statement via ``on_kill`` before the next
attempt gets a
-chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks`
for why.
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes the stored
+statement id by default from Airflow 3.4. Every other clear, such as ``airflow
tasks clear`` or
+clearing a whole Dag run, does so from Airflow 3.5. To keep it, tick **Keep
task state and resume**
+in the Clear dialog, or send ``"keep_task_state": true`` in the request. The
+**Keep task state on clear** UI setting ticks that box by default. From
Airflow 3.5,
+``airflow tasks clear --keep-task-state`` keeps it too. The next attempt then
reads the stored
+statement id back. In synchronous mode, what the next attempt does depends on
the state of the
+statement at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Statement state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running (the task had already failed, or its worker died)
+ - Submits the SQL again. The first statement keeps running.
+ - Reconnects to the running statement.
+ * - Succeeded
+ - Submits the SQL again.
+ - Returns right away without running the SQL again.
+ * - Failed or aborted
+ - Submits the SQL again.
+ - Submits the SQL again.
+
+With ``deferrable=True``, the operator does not store the statement id, so the
next attempt submits
+the SQL again whether or not task state was kept.
+
+With ``[state_store] clear_on_success`` on, the stored statement id is deleted
when the task
+succeeds, so clearing a succeeded task submits again even when task state is
kept.
+
+Keep task state when a statement is still running and nothing about the code
or inputs changed, so
+you don't pay for two. This assumes the statement survives the clear; keeping
task state does not
+prevent cancellation via ``on_kill``. See
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
+
+By default (``cancel_on_kill=True``), clearing a running task requests
cancellation via
+``on_kill``, including when deferred (Airflow 3.3+). With
``cancel_on_kill=False``, the statement is
+left running; a default clear submits the SQL again on the next synchronous
attempt.
To opt out and always submit fresh SQL on retry, set ``durable=False``:
diff --git a/providers/apache/spark/docs/operators.rst
b/providers/apache/spark/docs/operators.rst
index 92792a26725..18e32d56a86 100644
--- a/providers/apache/spark/docs/operators.rst
+++ b/providers/apache/spark/docs/operators.rst
@@ -223,15 +223,38 @@ prerequisite each one needs before the operator can check
driver status on retry
immediately) still emits a deprecation warning on every Airflow version
and maps onto
``durable``.
-Clearing a task is treated the same as a retry, which matters specifically for
a task whose driver
-already succeeded: clearing does not delete the stored driver ID, so the next
attempt reads it
-back and returns immediately without resubmitting. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
-polling synchronously can cancel the driver via ``on_kill`` before the next
attempt gets a chance
-to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for
why.
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes the stored
+driver ID by default from Airflow 3.4. Every other clear, such as ``airflow
tasks clear`` or
+clearing a whole Dag run, does so from Airflow 3.5. To keep it, tick **Keep
task state and resume**
+in the Clear dialog, or send ``"keep_task_state": true`` in the request. The
+**Keep task state on clear** UI setting ticks that box by default. From
Airflow 3.5,
+``airflow tasks clear --keep-task-state`` keeps it too. The next attempt then
reads the stored
+driver ID back. What the next attempt does depends on the state of the driver
at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Driver state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running (the task had already failed, or its worker died)
+ - Submits a new driver. The first one keeps running.
+ - Reconnects to the running driver.
+ * - Succeeded
+ - Submits a new driver.
+ - Returns right away without submitting another one.
+ * - Failed or killed
+ - Submits a new driver.
+ - Submits a new driver.
+
+With ``[state_store] clear_on_success`` on, the stored driver ID is deleted
when the task succeeds,
+so clearing a succeeded task submits again even when task state is kept.
+
+Keep task state when a driver is still running and nothing about the code or
inputs changed, so
+you don't pay for two. This assumes the driver survives the clear; keeping
task state does not
+prevent cancellation via ``on_kill``. See
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
.. _cluster-mode-crash-recovery-spark-standalone:
diff --git a/providers/common/ai/docs/durable_execution.rst
b/providers/common/ai/docs/durable_execution.rst
index abd0dda3c29..649ea1da0ad 100644
--- a/providers/common/ai/docs/durable_execution.rst
+++ b/providers/common/ai/docs/durable_execution.rst
@@ -414,16 +414,17 @@ rerun replays from the journal (see
:ref:`durable-clear`), the replayed steps co
Clearing a task
---------------
-On Airflow 3.4 and later, clearing a task instance (its Clear dialog in the
UI, or
-``POST /api/v2/dags/{dag_id}/clearTaskInstances``) deletes its task state, the
journal
-included, so the rerun starts the agent over. This holds for the default task
state store
-backend; a custom ``[state_store] backend`` keeps its entries. To resume from
the journal
-instead, tick **Keep task state and resume** in the dialog, or send
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes its
+task state by default from Airflow 3.4. Every other clear, such as
+``airflow tasks clear`` or clearing a whole Dag run, does so from Airflow 3.5.
The
+journal goes with it, so the rerun starts the agent over. This holds for the
default
+task state store backend; a custom ``[state_store] backend`` keeps its
entries. To keep
+task state, tick **Keep task state and resume** in the Clear dialog, or send
``"keep_task_state": true`` in the request. The **Keep task state on clear**
UI setting
-ticks that box by default. Clearing the whole Dag run from the Clear Run
dialog keeps task
-state; "Retries resume, clearing starts over" in
-:ref:`apache-airflow:concepts-resumable-tasks` lists the other ways of
clearing and which
-of them keep it.
+ticks that box by default. From Airflow 3.5, ``airflow tasks clear
--keep-task-state``
+keeps it too. The rerun then replays the journal and resumes the agent from
the last
+step it finished. "Retries resume, clearing starts over" in
+:ref:`apache-airflow:concepts-resumable-tasks` lists the ways of clearing.
Before Airflow 3.4 no clear deletes the journal (the task state store on 3.3,
the file
under ``durable_cache_path`` earlier), so a cleared task replays.
diff --git a/providers/common/ai/docs/frameworks/openai_agents.rst
b/providers/common/ai/docs/frameworks/openai_agents.rst
index 7a6b8cfcdd3..0d96b63c19a 100644
--- a/providers/common/ai/docs/frameworks/openai_agents.rst
+++ b/providers/common/ai/docs/frameworks/openai_agents.rst
@@ -705,10 +705,9 @@ Things to know:
a ``session``, when the call is made by an agent used through ``as_tool`` or
reached
through ``handoff()``, or when the approval is for a hosted MCP tool
(``HostedMCPTool``). After you remove the approval requirement, clearing the
task with
- its task state kept (``keep_task_state``, or clearing the whole Dag run)
replays the turns
- before that call and asks the model again from there, so a tool call made in
that same
- answer runs again if the model asks for it. A clear that discards the task
state starts
- the agent over.
+ its task state kept replays the turns before that call and asks the model
again from
+ there, so a tool call made in that same answer runs again if the model asks
for it. A
+ clear that discards the task state starts the agent over (see
:ref:`durable-clear`).
- Every pause carries the answers the task has received so far, so keep the
approvals one
run asks for to a handful.
diff --git a/providers/common/ai/docs/frameworks/strands.rst
b/providers/common/ai/docs/frameworks/strands.rst
index e7f7a9fff53..af3b8753db9 100644
--- a/providers/common/ai/docs/frameworks/strands.rst
+++ b/providers/common/ai/docs/frameworks/strands.rst
@@ -203,7 +203,7 @@ model failed in its second cycle; ``cycle`` counts from 0:
answer, or an interrupt or a cancellation, which it does not resume. It
deletes the
saved state then, so clearing the task later starts the agent over. If that
delete
fails, the task still succeeds and logs ``Could not delete the Strands agent's
saved
-state``, and a later clear of the whole Dag run resumes from the last pause.
+state``, and a later clear that keeps task state resumes from the last pause.
A retry runs some work again:
@@ -221,8 +221,9 @@ A retry runs some work again:
or keep sandbox work out of a durable agent.
A retry resumes the saved conversation with the system prompt it was saved
with.
-So does clearing the whole Dag run, which keeps task state. To run a changed
system
-prompt on a failed task, clear the task itself without keeping its task state.
+So does a clear that keeps task state. To run a changed system prompt on a
failed
+task, clear it without keeping its task state (see :ref:`durable-clear` for
when a
+clear keeps it).
An agent whose model keeps the conversation server-side (``model.stateful``)
raises
``ValueError``: Strands' `session manager
diff --git a/providers/databricks/docs/operators/run_now.rst
b/providers/databricks/docs/operators/run_now.rst
index 82a308f436e..325711b4c32 100644
--- a/providers/databricks/docs/operators/run_now.rst
+++ b/providers/databricks/docs/operators/run_now.rst
@@ -142,15 +142,45 @@ stored, so a retry that comes later than that finds
nothing and triggers a fresh
reconnecting, whatever schedule ``airflow state-store clean`` runs on. Size
that setting against
your longest ``retry_delay``.
-Clearing a task discards the stored run id by default, so the next attempt
triggers a fresh run.
-Set ``keep_task_state`` when clearing to keep it instead, and the attempt is
then treated the same
-as a retry: for a task whose run already succeeded, it reads the run back and
returns immediately
-without triggering a new one.
:doc:`apache-airflow:core-concepts/resumable-tasks` lists which
-clear paths discard the id and describes the ``[state_store]
clear_on_success`` setting.
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
-polling synchronously can cancel the run via ``on_kill`` before the next
attempt gets a chance to
-reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes the stored run
+id by default from Airflow 3.4. Every other clear, such as ``airflow tasks
clear`` or clearing a
+whole Dag run, does so from Airflow 3.5. To keep it, tick **Keep task state
and resume** in the
+Clear dialog, or send ``"keep_task_state": true`` in the request. The **Keep
task state on clear**
+UI setting ticks that box by default. From Airflow 3.5, ``airflow tasks clear
--keep-task-state``
+keeps it too. The next attempt then reads the stored run id back. In
synchronous mode, what the next
+attempt does depends on the state of the Databricks run at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Run state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running (the task had already failed, or its worker died)
+ - Triggers a new job run. The first one keeps running, unless
``cancel_previous_runs=True``
+ cancels it first.
+ - Reconnects to the running run.
+ * - Succeeded
+ - Triggers a new job run.
+ - Returns right away without triggering another one.
+ * - Failed or canceled
+ - Triggers a new job run.
+ - Triggers a new job run.
+
+With ``deferrable=True``, the operator does not store the run id, so the next
attempt triggers a
+new job run whether or not task state was kept.
+
+With ``[state_store] clear_on_success`` on, the stored run id is deleted when
the task succeeds, so
+clearing a succeeded task submits again even when task state is kept.
+
+Keep task state when a run is still running and nothing about the code or
inputs changed, so
+you don't pay for two. This assumes the run survives the clear; keeping task
state does not
+prevent cancellation via ``on_kill``. See
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
+
+Clearing a task that is still running cancels the run via ``on_kill``, also
when the task is
+deferred (Airflow 3.3+), so the next attempt finds it canceled.
To opt out and always trigger a fresh run on retry, set ``durable=False``:
diff --git a/providers/databricks/docs/operators/submit_run.rst
b/providers/databricks/docs/operators/submit_run.rst
index db23b2074d8..e3368b4586c 100644
--- a/providers/databricks/docs/operators/submit_run.rst
+++ b/providers/databricks/docs/operators/submit_run.rst
@@ -210,15 +210,44 @@ stored, so a retry that comes later than that finds
nothing and submits a fresh
reconnecting, whatever schedule ``airflow state-store clean`` runs on. Size
that setting against
your longest ``retry_delay``.
-Clearing a task discards the stored run id by default, so the next attempt
submits a fresh run.
-Set ``keep_task_state`` when clearing to keep it instead, and the attempt is
then treated the same
-as a retry: for a task whose run already succeeded, it reads the run back and
returns immediately
-without submitting a new one.
:doc:`apache-airflow:core-concepts/resumable-tasks` lists which
-clear paths discard the id and describes the ``[state_store]
clear_on_success`` setting.
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
-polling synchronously can cancel the run via ``on_kill`` before the next
attempt gets a chance to
-reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes the stored run
+id by default from Airflow 3.4. Every other clear, such as ``airflow tasks
clear`` or clearing a
+whole Dag run, does so from Airflow 3.5. To keep it, tick **Keep task state
and resume** in the
+Clear dialog, or send ``"keep_task_state": true`` in the request. The **Keep
task state on clear**
+UI setting ticks that box by default. From Airflow 3.5, ``airflow tasks clear
--keep-task-state``
+keeps it too. The next attempt then reads the stored run id back. In
synchronous mode, what the next
+attempt does depends on the state of the Databricks run at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Run state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running (the task had already failed, or its worker died)
+ - Submits a new run. The first one keeps running.
+ - Reconnects to the running run.
+ * - Succeeded
+ - Submits a new run.
+ - Returns right away without submitting another one.
+ * - Failed or canceled
+ - Submits a new run.
+ - Submits a new run.
+
+With ``deferrable=True``, the operator does not store the run id, so the next
attempt submits a new
+run whether or not task state was kept.
+
+With ``[state_store] clear_on_success`` on, the stored run id is deleted when
the task succeeds, so
+clearing a succeeded task submits again even when task state is kept.
+
+Keep task state when a run is still running and nothing about the code or
inputs changed, so
+you don't pay for two. This assumes the run survives the clear; keeping task
state does not
+prevent cancellation via ``on_kill``. See
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
+
+Clearing a task that is still running cancels the run via ``on_kill``, also
when the task is
+deferred (Airflow 3.3+), so the next attempt finds it canceled.
To opt out and always submit a fresh run on retry, set ``durable=False``:
diff --git a/providers/huggingface/docs/operators/datasets.rst
b/providers/huggingface/docs/operators/datasets.rst
index 345f00f61a3..f8e66310e25 100644
--- a/providers/huggingface/docs/operators/datasets.rst
+++ b/providers/huggingface/docs/operators/datasets.rst
@@ -129,9 +129,9 @@ Execution proceeds in this order:
config/revision, and pull-request setting. On Airflow 3.3 and later, it
checks
task state first. It searches Hub refs only when task state records that a
prior attempt may have started the push. A matching final commit recovers an
- interrupted publication. Clearing a task preserves that marker state on
- Airflow 3.3. On Airflow 3.4, clearing a Dag run also preserves it; clearing
- an individual task instance removes it only when ``keep_task_state`` is off.
+ interrupted publication. Clearing a task removes that marker state by
+ default: from the UI or the REST API from Airflow 3.4, and on every other
+ clear from Airflow 3.5. Set ``keep_task_state`` when clearing to keep it.
#. **Create a lazy stream if recovery found no commit.** The operator calls
``load_dataset(..., streaming=True)`` and passes the optional configuration,
split, and pinned revision. Hugging Face Datasets reads rows incrementally.
@@ -189,21 +189,23 @@ the same artifact to another target is a separate
publication. Airflow 3.3 and
later task state store support caches the result and records that a push may
have started. A retry searches the intended branch or pull request ref only
after that record exists, and only reuses the latest commit without a generated
-suffix carrying the marker. Clearing a task preserves this state on Airflow
-3.3. On Airflow 3.4, clearing a Dag run also preserves it; clearing an
-individual task instance removes it only when ``keep_task_state`` is off. In
-the cases that preserve state, a cleared task can return the stored commit and
-emit its asset event again without pushing. This avoids reusing an older
-publication after the branch has advanced. The task
-emits an asset event with the same commit SHA and URL only after successful
-publication. Publication uses Datasets ``push_to_hub``, which rewrites the
-README's YAML card metadata, including ``dataset_info`` and ``configs``, in the
-push commit. Only a publication to ``main`` without ``create_pr`` emits the
-repo-level asset event; pull request and non-main revision pushes do not change
-the revision referenced by ``hf://datasets/<repo_id>``. For those pushes the
-operator removes ``hf://datasets/<repo_id>`` from ``outlets`` even when you
list
-it there yourself, and in a mapped task this follows each expanded
-``push_kwargs`` entry.
+suffix carrying the marker. Clearing a task removes this state by default: from
+the UI or the REST API from Airflow 3.4, and on every other clear from Airflow
+3.5. If you set ``keep_task_state`` when clearing, a cleared task can return
+the stored commit and emit its asset event again without pushing. This avoids
+reusing an older publication after the branch has advanced. With
+``[state_store] clear_on_success`` on, the stored commit is deleted when the
+task succeeds, so clearing a succeeded task pushes again even when task state
+is kept. The task emits an asset event with the same commit SHA and URL only
+after successful publication. Publication uses Datasets ``push_to_hub``, which
+rewrites the README's YAML card metadata, including ``dataset_info`` and
+``configs``, in the push commit. Only a publication to ``main`` without
+``create_pr`` emits the repo-level asset event; pull request and non-main
+revision pushes do not change the revision referenced by
+``hf://datasets/<repo_id>``. For those pushes the operator removes
+``hf://datasets/<repo_id>`` from ``outlets`` even when you list it there
+yourself, and in a mapped task this follows each expanded ``push_kwargs``
+entry.
Example
-------
diff --git a/providers/snowflake/docs/operators/snowflake.rst
b/providers/snowflake/docs/operators/snowflake.rst
index 26cb1a2e7b8..ee9d9718a0b 100644
--- a/providers/snowflake/docs/operators/snowflake.rst
+++ b/providers/snowflake/docs/operators/snowflake.rst
@@ -189,15 +189,46 @@ between, the handles won't be there for the next retry,
and the operator will su
fresh instead of reconnecting. Avoid running cleanup on a schedule shorter
than your longest
``retry_delay``.
-Clearing a task is treated the same as a retry, which matters specifically for
a task whose
-statements already succeeded: clearing does not delete the stored handles, so
the next attempt
-reads them back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
-polling synchronously can cancel the statements via ``on_kill`` before the
next attempt gets a
-chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks`
for why.
+Clearing a task instance from the UI or the REST API (so ``airflowctl`` too)
deletes the stored
+statement handles by default from Airflow 3.4. Every other clear, such as
``airflow tasks clear`` or
+clearing a whole Dag run, does so from Airflow 3.5. To keep it, tick **Keep
task state and resume**
+in the Clear dialog, or send ``"keep_task_state": true`` in the request. The
+**Keep task state on clear** UI setting ticks that box by default. From
Airflow 3.5,
+``airflow tasks clear --keep-task-state`` keeps it too. The next attempt then
reads the stored
+statement handles back. In synchronous mode, what the next attempt does
depends on the state of the
+statements at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Statement state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running (the task had already failed, or its worker died)
+ - Submits the SQL again. The first statements keep running.
+ - Reconnects to the running statements.
+ * - Succeeded
+ - Submits the SQL again.
+ - Returns right away without running the SQL again.
+ * - Failed or aborted
+ - Submits the SQL again.
+ - Submits the SQL again.
+
+With ``deferrable=True``, the operator does not store the statement handles,
so the next attempt
+submits the SQL again whether or not task state was kept.
+
+With ``[state_store] clear_on_success`` on, the stored statement handles are
deleted when the task
+succeeds, so clearing a succeeded task submits again even when task state is
kept.
+
+Keep task state when statements are still running and nothing about the code
or inputs changed, so
+you don't pay for two. This assumes the statements survive the clear; keeping
task state does not
+prevent cancellation via ``on_kill``. See
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
+
+By default (``cancel_on_kill=True``), clearing a running task requests
cancellation via
+``on_kill``, including when deferred (Airflow 3.3+). With
``cancel_on_kill=False``, the statements
+are left running; a default clear submits the SQL again on the next
synchronous attempt.
To opt out and always submit fresh SQL on retry, set ``durable=False``: