RyujiTamaki commented on code in PR #77112: URL: https://github.com/apache/airflow/pull/77112#discussion_r4585416200
########## providers/google/src/airflow/providers/google/cloud/triggers/cloud_batch.py: ########## @@ -20,13 +20,34 @@ from collections.abc import AsyncIterator, Sequence from typing import Any +from google.api_core.exceptions import ( + Aborted, + DeadlineExceeded, + GatewayTimeout, + InternalServerError, + ResourceExhausted, + ServiceUnavailable, +) from google.cloud.batch_v1 import Job, JobStatus from airflow.providers.google.cloud.hooks.cloud_batch import CloudBatchAsyncHook from airflow.triggers.base import BaseTrigger, TriggerEvent DEFAULT_BATCH_LOCATION = "us-central1" +# gRPC errors that indicate a transient failure of the Cloud Batch API rather than a terminal +# state of the job. The default retry of ``BatchServiceAsyncClient.get_job`` only covers +# ServiceUnavailable, so the others would otherwise fail the deferred task while the job keeps +# running, and the task-level retry would submit the job again. +_RETRYABLE_GRPC_EXCEPTIONS = ( + ServiceUnavailable, # 503 / UNAVAILABLE + InternalServerError, # 500 / INTERNAL + DeadlineExceeded, # 504 / DEADLINE_EXCEEDED + GatewayTimeout, # 504 / gateway timeout + ResourceExhausted, # 429 / RESOURCE_EXHAUSTED + Aborted, # ABORTED +) Review Comment: @its-amann Thanks for catching this! You're right — I hadn't considered that get_job's default retry wraps a ServiceUnavailable that outlasts its 60s deadline in RetryError. I've applied your suggestion by cherry-picking your commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
