kaxil commented on code in PR #76968:
URL: https://github.com/apache/airflow/pull/76968#discussion_r4580833774


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,30 @@ id won't be there for the next retry, and the operator 
falls back to the XCom/sc
 above rather than reconnecting via task state store. Avoid running cleanup on 
a schedule shorter
 than your longest ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Glue job run at
+that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Job run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new job run. The first one keeps running.

Review Comment:
   With `deferrable=True` and the default `durable`, `submit_job` reattaches 
through the task UUID scan on any attempt after the first, without looking at 
the state store. So a default clear reconnects to a running Glue run here, it 
doesn't submit a new one (`resumable-tasks` says the same for a deferred 
`GlueJobOperator`). Should this table be scoped to the synchronous mode, with a 
line for the deferrable one?



##########
providers/databricks/docs/operators/run_now.rst:
##########
@@ -143,11 +143,30 @@ when someone runs ``airflow state-store clean``. If a 
task's ``retry_delay`` is
 run id won't be there for the next retry, and the operator will trigger a 
fresh run instead of
 reconnecting. Avoid running cleanup on a schedule shorter than your longest 
``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose run
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without triggering a new run. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Databricks run
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Triggers a new job run. The first one keeps running.

Review Comment:
   `on_kill` here cancels the run unconditionally, so clearing a task that is 
still polling stops the run instead of leaving it going. That is what the 
paragraph right under this table says, and `resumable-tasks` says it too. The 
first run only survives if the worker died without running `on_kill`, or the 
task had already failed. The same row is in `submit_run.rst`, `snowflake.rst`, 
`redshift_data.rst` and the Spark page, each with the same `on_kill` caveat 
below it. Can the row say when the old job is still alive?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to