amoghrajesh commented on code in PR #76413:
URL: https://github.com/apache/airflow/pull/76413#discussion_r4508685466
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:
##########
@@ -272,6 +272,10 @@ class SageMakerProcessingOperator(SageMakerBaseOperator):
{"output_name": "evaluation", "file_name": "metrics.json",
"result_name": "EvaluationReport"}
]
# Downstream:
ti.xcom_pull("this_task")["OutputFiles"]["EvaluationReport"]["mse"]
+ :param stop_processing_job_on_kill: If True, the processing job is stopped
when the task is killed.
+ Without it the job keeps running, and billing, after the task is gone.
A deferred task is
+ stopped from the Triggerer, which needs Airflow 3.3 or later; below
that the flag only
Review Comment:
`execute_complete` now stops the job on a "timeout" event when the flag is
set; "failed" and "stopped" are already terminal, so they are left alone. The
`self.defer(timeout=...)` expiry cannot be covered - the Triggerer kills the
trigger without an event, so nothing runs on the worker. Scoped to Processing
here; the training operator needs the same and is not touched by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
