shahar1 commented on issue #57359:
URL: https://github.com/apache/airflow/issues/57359#issuecomment-3858488161
> Hi - we have been using `apache-airflow-providers-google==19.3.0` for the
past several weeks and the issue recurred:
>
> ```
> [2026-02-02 04:42:52] ERROR - Exception occurred while checking for job
completion.
source=airflow.providers.google.cloud.triggers.dataflow.TemplateJobStartTrigger
loc=dataflow.py:149
> ServiceUnavailable: 503 Visibility check was unavailable. Please retry the
request and contact support if the problem persists
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/triggers/dataflow.py",
line 113 in run
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/dataflow.py",
line 1480 in get_job_status
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/dataflow.py",
line 1457 in get_job
> File
"/home/airflow/.local/lib/python3.12/site-packages/google/cloud/dataflow_v1beta3/services/jobs_v1_beta3/async_client.py",
line 478 in get_job
> File
"/home/airflow/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py",
line 88 in __await__
> AioRpcError: <AioRpcError of RPC that terminated with:
> status = StatusCode.UNAVAILABLE
> details = "Visibility check was unavailable. Please retry the request
and contact support if the problem persists"
> debug_error_string = "UNKNOWN:Error received from peer
ipv4:74.125.126.95:443 {created_time:"2026-02-02T12:42:52.166480427+00:00",
grpc_status:14, grpc_message:"Visibility check was unavailable. Please retry
the request and contact support if the problem persists"}"
> >
> File
"/home/airflow/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py",
line 85 in __await__
> File
"/home/airflow/.local/lib/python3.12/site-packages/grpc/aio/_interceptor.py",
line 472 in __await__
> File
"/home/airflow/.local/lib/python3.12/site-packages/grpc/aio/_call.py", line 327
in __await__
> ```
>
> The task retry error from the initial bug report `ValueError: dictionary
update sequence element #0 has length 1; 2 is required` no longer occurs but as
expected the task is still just considered failed since the triggerer marked
the task as failed, and the dataflow job is _not_ actually retried (nor does
the triggerer or any component attempt to get the status a second time)
>
> ```
> [2026-02-02 04:45:10] ERROR - Task failed with exception source=task
loc=task_runner.py:972
> AirflowException: 503 Visibility check was unavailable. Please retry the
request and contact support if the problem persists
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 920 in run
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 1307 in _execute_task
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py",
line 1632 in resume_execution
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/dataflow.py",
line 650 in execute_complete
> ```
>
> > manage to find how to reproduce it, please comment (without clear
reproduction steps there's not too much that we can do).
>
> it is dependent on the GCP dataflow api returning a 503 - the only way to
reliably reproduce it would be with a mock http server that the gcp client
connects to and having it return a 503 (or by mocking out the
`JobsV1Beta3AsyncClient.get_job` method to throw the AioRpcError in the
stacktrace above). i don't have permissions to reopen this issue but it is
definitely a still a bug with the retry logic in the provider
I've reopened it - if you or someone else could implement the mock for
reproducing the issue, it would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
