its-amann commented on code in PR #77112:
URL: https://github.com/apache/airflow/pull/77112#discussion_r4582050588


##########
providers/google/src/airflow/providers/google/cloud/triggers/cloud_batch.py:
##########
@@ -20,13 +20,34 @@
 from collections.abc import AsyncIterator, Sequence
 from typing import Any
 
+from google.api_core.exceptions import (
+    Aborted,
+    DeadlineExceeded,
+    GatewayTimeout,
+    InternalServerError,
+    ResourceExhausted,
+    ServiceUnavailable,
+)
 from google.cloud.batch_v1 import Job, JobStatus
 
 from airflow.providers.google.cloud.hooks.cloud_batch import 
CloudBatchAsyncHook
 from airflow.triggers.base import BaseTrigger, TriggerEvent
 
 DEFAULT_BATCH_LOCATION = "us-central1"
 
+# gRPC errors that indicate a transient failure of the Cloud Batch API rather 
than a terminal
+# state of the job. The default retry of ``BatchServiceAsyncClient.get_job`` 
only covers
+# ServiceUnavailable, so the others would otherwise fail the deferred task 
while the job keeps
+# running, and the task-level retry would submit the job again.
+_RETRYABLE_GRPC_EXCEPTIONS = (
+    ServiceUnavailable,  # 503 / UNAVAILABLE
+    InternalServerError,  # 500 / INTERNAL
+    DeadlineExceeded,  # 504 / DEADLINE_EXCEEDED
+    GatewayTimeout,  # 504 / gateway timeout
+    ResourceExhausted,  # 429 / RESOURCE_EXHAUSTED
+    Aborted,  # ABORTED
+)

Review Comment:
   the SDK's default retry on get_job already retries ServiceUnavailable for 
60s and then raises RetryError (not ServiceUnavailable), so a 503 outage longer 
than that still skips this branch and fails the task. catching RetryError too 
covers it (needs `RetryError` added to the `google.api_core.exceptions` import 
above):
   
   ```suggestion
       Aborted,  # ABORTED
       # get_job's default retry wraps a ServiceUnavailable that outlasts its 
60s deadline in RetryError.
       RetryError,
   )
   ```
   
   fixed this plus a test in this commit, you can just cherry pick it instead 
of redoing it: 
https://github.com/its-amann/airflow/commit/85249ded25902a1be04e0bc1db8beee5e3201110
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to