aaron-y-chen commented on code in PR #76968:
URL: https://github.com/apache/airflow/pull/76968#discussion_r4589679497


##########
providers/amazon/docs/operators/redshift/redshift_data.rst:
##########
@@ -99,15 +99,35 @@ the statement id won't be there for the next retry, and the 
operator will submit
 instead of reconnecting. Avoid running cleanup on a schedule shorter than your 
longest
 ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose
-statement already succeeded: clearing does not delete the stored statement id, 
so the next attempt
-reads it back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a 
task that's actively
-polling synchronously can cancel the statement via ``on_kill`` before the next 
attempt gets a
-chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` 
for why.
+Clearing a task deletes the stored statement id by default (Airflow 3.4+), and 
so does clearing its
+whole Dag run (Airflow 3.5+). You can choose to keep task state when you 
clear, and the next attempt
+then reads the stored statement id back. What the next attempt does depends on 
the state of the
+statement at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Statement state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running (the task had already failed, or its worker died)
+     - Submits the SQL again. The first statement keeps running.
+     - Reconnects to the running statement.
+   * - Succeeded
+     - Submits the SQL again.
+     - Returns right away without running the SQL again.
+   * - Failed or aborted
+     - Submits the SQL again.
+     - Submits the SQL again.
+
+Keep task state when a statement is still running and nothing about the code 
or inputs changed, so
+you don't pay for two. This assumes the statement survives the clear; keeping 
task state does not
+prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
+
+Clearing a task that is still running cancels the statement via ``on_kill``, 
also when the task is
+deferred, so the next attempt finds it canceled.

Review Comment:
   ```suggestion
   By default (``cancel_on_kill=True``), clearing a running task requests 
cancellation via
   ``on_kill``, including when deferred. With ``cancel_on_kill=False``, the 
statement is left
   running; a default clear submits the SQL again on the next synchronous 
attempt.
   ```



##########
providers/snowflake/docs/operators/snowflake.rst:
##########
@@ -189,15 +189,35 @@ between, the handles won't be there for the next retry, 
and the operator will su
 fresh instead of reconnecting. Avoid running cleanup on a schedule shorter 
than your longest
 ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose
-statements already succeeded: clearing does not delete the stored handles, so 
the next attempt
-reads them back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a 
task that's actively
-polling synchronously can cancel the statements via ``on_kill`` before the 
next attempt gets a
-chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` 
for why.
+Clearing a task deletes the stored statement handles by default (Airflow 
3.4+), and so does clearing
+its whole Dag run (Airflow 3.5+). You can choose to keep task state when you 
clear, and the next
+attempt then reads the stored statement handles back. What the next attempt 
does depends on the
+state of the statements at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Statement state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running (the task had already failed, or its worker died)
+     - Submits the SQL again. The first statements keep running.
+     - Reconnects to the running statements.
+   * - Succeeded
+     - Submits the SQL again.
+     - Returns right away without running the SQL again.
+   * - Failed or aborted
+     - Submits the SQL again.
+     - Submits the SQL again.
+
+Keep task state when statements are still running and nothing about the code 
or inputs changed, so
+you don't pay for two. This assumes the statements survive the clear; keeping 
task state does not
+prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
+for the ways of clearing.
+
+Clearing a task that is still running cancels the statements via ``on_kill``, 
also when the task is
+deferred, so the next attempt finds it canceled.

Review Comment:
   ```suggestion
   By default (``cancel_on_kill=True``), clearing a running task requests 
cancellation via
   ``on_kill``, including when deferred. With ``cancel_on_kill=False``, the 
statements are left
   running; a default clear submits the SQL again on the next synchronous 
attempt.
   ```



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,33 @@ id won't be there for the next retry, and the operator 
falls back to the XCom/sc
 above rather than reconnecting via task state store. Avoid running cleanup on 
a schedule shorter
 than your longest ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. In synchronous mode, what the next attempt does 
depends on the state
+of the Glue job run at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Job run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new job run. The first one keeps running.
+     - Reconnects to the running job run.
+   * - Succeeded
+     - Submits a new job run.
+     - Returns right away without starting another one.
+   * - Failed or stopped
+     - Submits a new job run.
+     - Submits a new job run.
+
+With ``deferrable=True``, the next attempt finds a still-running job run and 
reconnects to it on any
+clear, whether or not task state was kept.

Review Comment:
   IIUC, reconnection here also depends on the stop policy. WDYT?
   
   ```suggestion
   With ``deferrable=True``, ``durable=True`` and 
``stop_job_run_on_kill=False`` (the default), the
   next attempt reconnects to a still-running job run on clear, whether or not 
task state was kept.
   With ``stop_job_run_on_kill=True`` instead, it stops the previous run before 
submitting a new one.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to