kaxil commented on code in PR #76964:
URL: https://github.com/apache/airflow/pull/76964#discussion_r4580733857


##########
providers/amazon/docs/operators/sagemaker.rst:
##########
@@ -67,6 +67,51 @@ To create an Amazon Sagemaker training job you can use
     :start-after: [START howto_operator_sagemaker_training]
     :end-before: [END howto_operator_sagemaker_training]
 
+Durable execution
+"""""""""""""""""
+
+``SageMakerTrainingOperator`` submits a training job and then polls it to 
completion on the
+worker. By default the operator runs in a *durable* mode that makes this 
crash-safe: the training
+job name is persisted to :doc:`task state store 
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task 
is retried, the
+operator reconnects to the job that is already running in SageMaker instead of 
starting a new one.
+
+On retry the operator checks the prior job's status:
+
+* if it is ``InProgress``, the operator reconnects and continues polling
+* if it is ``Completed``, the operator returns its result without resubmitting
+* if it is ``Failed``, ``Stopped`` or ``Stopping``, or the name is no longer 
found, the operator
+  submits a new job
+
+Unlike Glue, a job that is still ``Stopping`` gets a new submit rather than a 
reconnect. A stopping
+SageMaker job never produces a model, and ``cancel_on_kill`` leaves the job in 
exactly that state
+when a timed out attempt is killed. A new submit is renamed with a timestamp 
suffix when
+``action_if_job_exists="timestamp"`` (the default), because the earlier job 
still holds its name.
+Spot training resume through ``CheckpointConfig`` is unaffected, since the 
checkpoint
+``S3Uri`` does not depend on the job name.
+
+Clearing a task is treated the same as a retry, which matters for a task whose 
job already
+completed: clearing does not delete the stored job name, so the next attempt 
returns the existing
+result without starting a new job. See 
:doc:`apache-airflow:core-concepts/resumable-tasks` for why,

Review Comment:
   This only holds on 3.3. From 3.4, clearing a single task instance (UI, REST 
API, `airflowctl`) discards the task state store unless `keep_task_state` is 
set, so the next attempt submits a new training job. Clearing a whole Dag run, 
mark failed/success and `airflow tasks clear` still keep it. The linked 
`resumable-tasks` page now says clearing starts over, so this contradicts the 
page it cites. Could the paragraph split by version and point at 
`keep_task_state` for reconnecting? The Glue page has the same paragraph.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to