dondaum commented on issue #38532: URL: https://github.com/apache/airflow/issues/38532#issuecomment-2049580920
I tried to reproduce the exact error but with no success. I tried to reproduce it with the following DAG: ```Python import datetime import os from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator from airflow import DAG WAIT_QUERY = """ DECLARE retry_count INT64; DECLARE success BOOL; DECLARE size_bytes INT64; DECLARE row_count INT64; DECLARE DELAY_TIME DATETIME; DECLARE WAIT STRING; SET retry_count = 2; SET success = FALSE; WHILE retry_count <= 3 AND success = FALSE DO BEGIN SET row_count = (with a as (SELECT 1 as b) SELECT * FROM a WHERE 1 = 2); IF row_count > 0 THEN SELECT 'Table Exists!' as message, retry_count as retries; SET success = TRUE; ELSE SELECT 'Table does not exist' as message, retry_count as retries, row_count; SET retry_count = retry_count + 1; -- WAITFOR DELAY '00:00:10'; SET WAIT = 'TRUE'; SET DELAY_TIME = DATETIME_ADD(CURRENT_DATETIME,INTERVAL 90 SECOND); WHILE WAIT = 'TRUE' DO IF (DELAY_TIME < CURRENT_DATETIME) THEN SET WAIT = 'FALSE'; END IF; END WHILE; END IF; END; END WHILE; """ with DAG( dag_id=os.path.splitext(os.path.basename(__file__))[0], schedule=None, start_date=datetime.datetime(2024, 1, 1), catchup=False, tags=["testing"], ) as dag: for i in range(10): bq_task = BigQueryInsertJobOperator( task_id=f"debug_query_{i}", configuration={ "query": { "query": WAIT_QUERY, "useLegacySql": False, "priority": "BATCH", } }, location="europe-west3", deferrable=True, ) ``` Also, I set the retry option in the GCP connection to 0 so as not to implicitly retry on failure. Could you perhaps create a DAG that reproduces the error? And maybe you could also check which apache-airflow-providers-google you are using? My setup: ``` Apache Airflow version | 2.7.3 executor | LocalExecutor task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler sql_alchemy_conn | postgresql+psycopg2://airflow:airflow@postgres/airflow dags_folder | /opt/airflow/dags plugins_folder | /opt/airflow/plugins base_log_folder | /opt/airflow/logs remote_base_log_folder | System info OS | Linux architecture | x86_64 uname | uname_result(system='Linux', node='42d8cf034cfc', release='5.10.16.3-microsoft-standard-WSL2', version='#1 SMP Fri Apr 2 22:23:49 UTC 2021', machine='x86_64') locale | ('en_US', 'UTF-8') python_version | 3.11.6 (main, Nov 1 2023, 14:02:22) [GCC 10.2.1 20210110] python_location | /usr/local/bin/python Tools info git | NOT AVAILABLE ssh | OpenSSH_8.4p1 Debian-5+deb11u2, OpenSSL 1.1.1w 11 Sep 2023 kubectl | NOT AVAILABLE gcloud | NOT AVAILABLE cloud_sql_proxy | NOT AVAILABLE mysql | mysql Ver 8.0.35 for Linux on x86_64 (MySQL Community Server - GPL) sqlite3 | 3.34.1 2021-01-20 14:10:07 10e20c0b43500cfb9bbc0eaa061c57514f715d87238f4d835880cd846b9ealt1 psql | psql (PostgreSQL) 16.0 (Debian 16.0-1.pgdg110+1) Paths info airflow_home | /opt/airflow system_path | /root/bin:/home/airflow/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin python_path | /home/airflow/.local/bin:/usr/local/lib/python311.zip:/usr/local/lib/python3.11:/usr/local/lib/python3.11/lib-dynload:/home/airflow/.local/lib/python3.11/site-packages: | /usr/local/lib/python3.11/site-packages:/opt/airflow/dags:/opt/airflow/config:/opt/airflow/plugins airflow_on_path | True Providers info apache-airflow-providers-amazon | 8.10.0 apache-airflow-providers-apache-beam | 5.6.2 apache-airflow-providers-celery | 3.6.0 apache-airflow-providers-cncf-kubernetes | 8.0.1 apache-airflow-providers-common-sql | 1.11.1 apache-airflow-providers-daskexecutor | 1.1.0 apache-airflow-providers-dbt-cloud | 3.7.0 apache-airflow-providers-docker | 3.8.0 apache-airflow-providers-elasticsearch | 5.1.0 apache-airflow-providers-ftp | 3.7.0 apache-airflow-providers-google | 10.16.0 apache-airflow-providers-grpc | 3.3.0 apache-airflow-providers-hashicorp | 3.6.4 apache-airflow-providers-http | 4.10.0 apache-airflow-providers-imap | 3.5.0 apache-airflow-providers-microsoft-azure | 8.1.0 apache-airflow-providers-mysql | 5.5.4 apache-airflow-providers-odbc | 4.1.0 apache-airflow-providers-openlineage | 1.2.0 apache-airflow-providers-postgres | 5.10.2 apache-airflow-providers-redis | 3.4.0 apache-airflow-providers-sendgrid | 3.4.0 apache-airflow-providers-sftp | 4.9.0 apache-airflow-providers-slack | 8.3.0 apache-airflow-providers-snowflake | 5.1.0 apache-airflow-providers-sqlite | 3.7.1 apache-airflow-providers-ssh | 3.10.1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org