kaxil commented on code in PR #76968: URL: https://github.com/apache/airflow/pull/76968#discussion_r4580833774
########## providers/amazon/docs/operators/glue.rst: ########## @@ -203,11 +203,30 @@ id won't be there for the next retry, and the operator falls back to the XCom/sc above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``. -Clearing a task is treated the same as a retry, which matters specifically for a task whose job -already succeeded: clearing does not delete the stored run id, so the next attempt reads it back -and returns immediately without submitting anything to Glue. See -:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the -``[state_store] clear_on_success`` setting that restores "clearing always resubmits." +Clearing a task deletes the stored run id by default (Airflow 3.4+), and so does clearing its whole +Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and the next attempt then +reads the stored run id back. What the next attempt does depends on the state of the Glue job run at +that point: + +.. list-table:: + :header-rows: 1 + :widths: 20 40 40 + + * - Job run state + - Default clear + - Clear that keeps task state + * - Still running + - Submits a new job run. The first one keeps running. Review Comment: With `deferrable=True` and the default `durable`, `submit_job` reattaches through the task UUID scan on any attempt after the first, without looking at the state store. So a default clear reconnects to a running Glue run here, it doesn't submit a new one (`resumable-tasks` says the same for a deferred `GlueJobOperator`). Should this table be scoped to the synchronous mode, with a line for the deferrable one? ########## providers/databricks/docs/operators/run_now.rst: ########## @@ -143,11 +143,30 @@ when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is run id won't be there for the next retry, and the operator will trigger a fresh run instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``. -Clearing a task is treated the same as a retry, which matters specifically for a task whose run -already succeeded: clearing does not delete the stored run id, so the next attempt reads it back -and returns immediately without triggering a new run. See -:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the -``[state_store] clear_on_success`` setting that restores "clearing always resubmits." +Clearing a task deletes the stored run id by default (Airflow 3.4+), and so does clearing its whole +Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and the next attempt then +reads the stored run id back. What the next attempt does depends on the state of the Databricks run +at that point: + +.. list-table:: + :header-rows: 1 + :widths: 20 40 40 + + * - Run state + - Default clear + - Clear that keeps task state + * - Still running + - Triggers a new job run. The first one keeps running. Review Comment: `on_kill` here cancels the run unconditionally, so clearing a task that is still polling stops the run instead of leaving it going. That is what the paragraph right under this table says, and `resumable-tasks` says it too. The first run only survives if the worker died without running `on_kill`, or the task had already failed. The same row is in `submit_run.rst`, `snowflake.rst`, `redshift_data.rst` and the Spark page, each with the same `on_kill` caveat below it. Can the row say when the old job is still alive? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
