Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
eladkal merged PR #66962: URL: https://github.com/apache/airflow/pull/66962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on code in PR #66962:
URL: https://github.com/apache/airflow/pull/66962#discussion_r3243617511
##
providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py:
##
@@ -712,11 +712,12 @@ def insert_all(
:param project_id: The name of the project where we have the table
:param dataset_id: The name of the dataset where we have the table
:param table_id: The name of the table
-:param rows: the rows to insert
+:param rows: the rows to insert. Each row is a mapping of column name
to
+value, matching the table schema.
.. code-block:: python
-rows = [{"json": {"a_key": "a_value_0"}}, {"json": {"a_key":
"a_value_1"}}]
Review Comment:
This format seems to have been wrong for quite a while (since 2020)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on code in PR #66962:
URL: https://github.com/apache/airflow/pull/66962#discussion_r3243617511
##
providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py:
##
@@ -712,11 +712,12 @@ def insert_all(
:param project_id: The name of the project where we have the table
:param dataset_id: The name of the dataset where we have the table
:param table_id: The name of the table
-:param rows: the rows to insert
+:param rows: the rows to insert. Each row is a mapping of column name
to
+value, matching the table schema.
.. code-block:: python
-rows = [{"json": {"a_key": "a_value_0"}}, {"json": {"a_key":
"a_value_1"}}]
Review Comment:
This syntax seems to have been wrong for quite a while
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on code in PR #66962: URL: https://github.com/apache/airflow/pull/66962#discussion_r3243619729 ## providers/google/src/airflow/providers/google/cloud/sensors/bigquery.py: ## @@ -332,6 +333,14 @@ class BigQueryStreamingBufferEmptySensor(BaseSensorOperator): ``UPDATE/MERGE/DELETE statement over table ... would affect rows in the streaming buffer`` errors. +.. warning:: +The sensor reads ``table.streaming_buffer`` from BigQuery's table +metadata, which is eventually consistent. For a short window right +after a streaming insert the buffer metadata is still absent, so the +sensor may report the buffer empty before it actually is. Known +limitation tracked at +https://github.com/apache/airflow/issues/66963 Review Comment: Flaw in the existing design, can be fixed later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Fix BigQueryStreamingBufferEmptySensor failing against real BigQuery [airflow]
shahar1 commented on PR #66962: URL: https://github.com/apache/airflow/pull/66962#issuecomment-4453834394 CC: @radhwene @VladaZakharova @MaksYermak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
