anxkhn opened a new pull request, #69118: URL: https://github.com/apache/airflow/pull/69118
BigQuery's `streamingBuffer` table metadata is eventually consistent: for several seconds after a streaming insert the rows are physically in the buffer but the metadata still reads absent. `BigQueryStreamingBufferEmptySensor` (added in #66652) decided "buffer empty" from a single absent reading, so during that window it reported empty too early and a downstream DML task (UPDATE / DELETE / MERGE) hit the very `... would affect rows in the streaming buffer` error the sensor exists to prevent. This makes "empty" trustworthy by requiring `empty_confirmations` consecutive empty readings (default `2`), each one `poke_interval` apart, before the sensor succeeds. A non-empty reading resets the counter. The same rule is applied in both the sync `poke()` path and the deferrable `BigQueryStreamingBufferEmptyTrigger`. Why consecutive confirmations instead of a non-empty -> empty transition check (as discussed in the issue thread): a transition check polls until timeout when the buffer flushes between two polls, and never fires for a table that is genuinely empty (no prior streaming insert). A bounded consecutive-empty count spans the consistency window while still terminating in both of those cases. The default of `2` forces one full `poke_interval` to elapse between the two empty readings, comfortably covering the ~10-12s metadata-lag window, while remaining configurable (validated `>= 1`). The obsolete metadata-wait workaround in the system test (`example_bigquery_streaming_buffer_sensor.py`) is removed, as #66963 requested once the sensor handles this itself. closes: #66963 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: OpenCode (Claude Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
