amoghrajesh commented on code in PR #76968:
URL: https://github.com/apache/airflow/pull/76968#discussion_r4614397460
##########
providers/databricks/docs/operators/run_now.rst:
##########
@@ -142,15 +142,35 @@ stored, so a retry that comes later than that finds
nothing and triggers a fresh
reconnecting, whatever schedule ``airflow state-store clean`` runs on. Size
that setting against
your longest ``retry_delay``.
-Clearing a task discards the stored run id by default, so the next attempt
triggers a fresh run.
-Set ``keep_task_state`` when clearing to keep it instead, and the attempt is
then treated the same
-as a retry: for a task whose run already succeeded, it reads the run back and
returns immediately
-without triggering a new one.
:doc:`apache-airflow:core-concepts/resumable-tasks` lists which
-clear paths discard the id and describes the ``[state_store]
clear_on_success`` setting.
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
-polling synchronously can cancel the run via ``on_kill`` before the next
attempt gets a chance to
-reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so
does clearing its whole
+Dag run (Airflow 3.5+). Set ``keep_task_state`` when clearing to keep it, and
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state
of the Databricks run
+at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Run state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running (the task had already failed, or its worker died)
+ - Triggers a new job run. The first one keeps running.
Review Comment:
2e28d15b392
##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,34 @@ id won't be there for the next retry, and the operator
falls back to the XCom/sc
above rather than reconnecting via task state store. Avoid running cleanup on
a schedule shorter
than your longest ``retry_delay``.
-Clearing a task is treated the same as a retry, which matters specifically for
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and
the next attempt then
+reads the stored run id back. In synchronous mode, what the next attempt does
depends on the state
+of the Glue job run at that point:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 20 40 40
+
+ * - Job run state
+ - Default clear
+ - Clear that keeps task state
+ * - Still running
+ - Submits a new job run. The first one keeps running.
+ - Reconnects to the running job run.
+ * - Succeeded
+ - Submits a new job run.
+ - Returns right away without starting another one.
Review Comment:
2e28d15b392
##########
providers/common/ai/docs/durable_execution.rst:
##########
@@ -414,16 +414,13 @@ rerun replays from the journal (see
:ref:`durable-clear`), the replayed steps co
Clearing a task
---------------
-On Airflow 3.4 and later, clearing a task instance (its Clear dialog in the
UI, or
-``POST /api/v2/dags/{dag_id}/clearTaskInstances``) deletes its task state, the
journal
-included, so the rerun starts the agent over. This holds for the default task
state store
-backend; a custom ``[state_store] backend`` keeps its entries. To resume from
the journal
-instead, tick **Keep task state and resume** in the dialog, or send
-``"keep_task_state": true`` in the request. The **Keep task state on clear**
UI setting
-ticks that box by default. Clearing the whole Dag run from the Clear Run
dialog keeps task
-state; "Retries resume, clearing starts over" in
-:ref:`apache-airflow:concepts-resumable-tasks` lists the other ways of
clearing and which
-of them keep it.
+Clearing a task deletes its task state by default (Airflow 3.4+), and so does
clearing
+its whole Dag run (Airflow 3.5+). The journal goes with it, so the rerun
starts the agent
+over. This holds for the default task state store backend; a custom
+``[state_store] backend`` keeps its entries. You can choose to keep task state
when you
Review Comment:
2e28d15b392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]