roykoand opened a new issue, #73981: URL: https://github.com/apache/airflow/issues/73981
### Under which category would you file this issue? Providers ### Apache Airflow version main (dev) ### What happened and how to reproduce it? In deferrable mode, `BigQueryValueCheckOperator` and `BigQueryIntervalCheckOperator` fail with the message `Job running` while the BigQuery job is still executing normally. `BigQueryAsyncHook.get_job_status()` returns the job state as-is for any job that isn't `DONE`, so a job in progress reports either `"pending"` or `"running"`: https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L2381-L2385 Both triggers treat only `"pending"` as "still in progress" and send every other status, including `"running"`, to the error branch: `BigQueryValueCheckTrigger.run()`: https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py#L687-L695 `BigQueryIntervalCheckTrigger.run()`: https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py#L584-L590 Repro: 1. Use `BigQueryValueCheckOperator(..., deferrable=True)` (or `BigQueryIntervalCheckOperator`) with a query that takes longer than one `poll_interval`. 2. The job moves to `RUNNING`, and the trigger yields `{"status": "error", "message": "Job running"}`. 3. The task fails, even though the query would have succeeded. There is a second problem in `BigQueryIntervalCheckTrigger`: the error event always uses the second job's message: https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py#L591-L595 When the first job fails and the second succeeds, the task fails with `Job completed` and the real error is lost. The other triggers in the same file use `if success / elif error / else keep polling`, so they don't have this problem. For example, `BigQueryInsertJobTrigger.run()`: https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/src/airflow/providers/google/cloud/triggers/bigquery.py#L187-L212 Existing tests only mock the `"pending"` status, never `"running"`, which is why this wasn't caught: https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/tests/unit/google/cloud/triggers/test_bigquery.py#L720-L724 https://github.com/apache/airflow/blob/e92147c12aae3877b4842d7eb8de33233cb6d45a/providers/google/tests/unit/google/cloud/triggers/test_bigquery.py#L816-L820 ### What you think should happen instead? Any status that is neither `success` nor `error` (including `running`) should keep polling, matching the other triggers in the same module. `BigQueryIntervalCheckTrigger` should report the message of the job that actually failed. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers apache-airflow-providers-google 22.5.0 ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
