Copilot commented on code in PR #64745:
URL: https://github.com/apache/airflow/pull/64745#discussion_r3068402865
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -368,33 +381,28 @@ def monitor_job(self, context: Context):
job_queue_arn=job_queue_arn,
)
- if self.awslogs_enabled:
- if self.waiters:
- self.waiters.wait_for_job(self.job_id,
get_batch_log_fetcher=self._get_batch_log_fetcher)
- else:
- self.hook.wait_for_job(self.job_id,
get_batch_log_fetcher=self._get_batch_log_fetcher)
- else:
- if self.waiters:
- self.waiters.wait_for_job(self.job_id)
- else:
- self.hook.wait_for_job(self.job_id)
+ return job_desc
- awslogs = []
+ def _persist_cloudwatch_link(self, context: Context) -> None:
+ """
+ Persist CloudWatch logs link if available.
+
+ :param context: Task context
+ """
Review Comment:
`_persist_cloudwatch_link()` always calls `get_job_all_awslogs_info()`
(which triggers a `describe_jobs`) even when `do_xcom_push=False`, but in that
case `CloudWatchEventsLink.persist()` will no-op and the API call is wasted.
Consider returning early in `_persist_cloudwatch_link()` when `not
self.do_xcom_push` to avoid unnecessary AWS API usage.
```suggestion
"""
if not self.do_xcom_push:
return
```
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -257,9 +263,9 @@ def execute_complete(self, context: Context, event:
dict[str, Any] | None = None
self.job_id = validated_event["job_id"]
Review Comment:
In `execute_complete()`, the early `status != "success"` check raises before
`self.job_id` is set, so the operator cannot persist the CloudWatch operator
link for deferred *failures*. This makes deferrable behavior inconsistent with
`monitor_job()` (which persists the CloudWatch link and then raises on
failure). Set `self.job_id` from the event before the status check, and persist
the CloudWatch link (when possible) even on failure before raising the
exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
