kandharvishnuu opened a new issue, #40837: URL: https://github.com/apache/airflow/issues/40837
### Apache Airflow Provider(s) google ### Versions of Apache Airflow Providers apache-airflow-providers-google==10.12.0 ### Apache Airflow version 2.7.3 ### Operating System debian ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### What happened When the [BigQueryInsertJobOperator](https://github.com/apache/airflow/blob/4298c433bc8b3136b8131ebe006dcf10c601e9c1/airflow/providers/google/cloud/operators/bigquery.py#L2654) submits a job to BigQuery, there are instances where the TaskInstance in Airflow is marked as `success` before the task actually completes in BigQuery. When the `execute` method of a TaskInstance’s Operator completes without raising an exception, the TaskInstance is marked as `success`. Since these TaskInstances are being marked as `success`, it implies that the BigQueryInsertJobOperator’s `execute` method is completing without any exceptions. When [QueryJob.result](https://github.com/googleapis/python-bigquery/blob/49bfd124cc76a719acf9257610181612d1452e56/google/cloud/bigquery/job/query.py#L1463) returns _something_, allowing the `execute` method to finish. ### What you think should happen instead We need to add a check [here](https://github.com/apache/airflow/blob/4298c433bc8b3136b8131ebe006dcf10c601e9c1/airflow/providers/google/cloud/operators/bigquery.py#L2866) to verify if the Job state is `DONE` in BigQuery, ensuring that Airflow only marks the TaskInstance as `success` when the job is indeed complete in BigQuery. ### How to reproduce I couldn't reproduce the issue. It occurs very rarely ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
