RyujiTamaki commented on code in PR #77112:
URL: https://github.com/apache/airflow/pull/77112#discussion_r4585416200


##########
providers/google/src/airflow/providers/google/cloud/triggers/cloud_batch.py:
##########
@@ -20,13 +20,34 @@
 from collections.abc import AsyncIterator, Sequence
 from typing import Any
 
+from google.api_core.exceptions import (
+    Aborted,
+    DeadlineExceeded,
+    GatewayTimeout,
+    InternalServerError,
+    ResourceExhausted,
+    ServiceUnavailable,
+)
 from google.cloud.batch_v1 import Job, JobStatus
 
 from airflow.providers.google.cloud.hooks.cloud_batch import 
CloudBatchAsyncHook
 from airflow.triggers.base import BaseTrigger, TriggerEvent
 
 DEFAULT_BATCH_LOCATION = "us-central1"
 
+# gRPC errors that indicate a transient failure of the Cloud Batch API rather 
than a terminal
+# state of the job. The default retry of ``BatchServiceAsyncClient.get_job`` 
only covers
+# ServiceUnavailable, so the others would otherwise fail the deferred task 
while the job keeps
+# running, and the task-level retry would submit the job again.
+_RETRYABLE_GRPC_EXCEPTIONS = (
+    ServiceUnavailable,  # 503 / UNAVAILABLE
+    InternalServerError,  # 500 / INTERNAL
+    DeadlineExceeded,  # 504 / DEADLINE_EXCEEDED
+    GatewayTimeout,  # 504 / gateway timeout
+    ResourceExhausted,  # 429 / RESOURCE_EXHAUSTED
+    Aborted,  # ABORTED
+)

Review Comment:
   @its-amann 
   
   Thanks for catching this! You're right — I hadn't considered that get_job's 
default retry wraps a ServiceUnavailable that outlasts its 60s deadline in 
RetryError. I've applied your suggestion by cherry-picking your commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to