ahmedabu98 commented on code in PR #26889: URL: https://github.com/apache/beam/pull/26889#discussion_r1212898791
########## website/www/site/content/en/documentation/io/built-in/google-bigquery.md: ########## @@ -779,6 +779,17 @@ Starting with version 2.36.0 of the Beam SDK for Java, you can use the [BigQuery Storage Write API](https://cloud.google.com/bigquery/docs/write-api) from the BigQueryIO connector. +Also after version 2.47.0 of Beam SDK for Python, SDK supports BigQuery Storage Write API. + +{{< paragraph class="language-py" >}} +BigQuery Storage Write API for Python SDK has some limitations on supported data types. This is due to that the Python SDK uses cross-language transforms for the feature. For example, `decimal.Decimal` in Python is needed to write a `NUMERIC` type in BigQuery. Also `DATETIME` in Python is not supported yet. For more details, please refer to the [type mapping](https://github.com/apache/beam/blob/0b430748cdd2e25edc553747ce018195e9cce888/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L112-L123). Review Comment: ```suggestion BigQuery Storage Write API for Python SDK currently has some limitations on supported data types. As this method makes use of cross-language transforms, we are limited to the types supported at the cross-language boundary. For example, `apache_beam.utils.timestamp.Timestamp` is needed to write a `TIMESTAMP` BigQuery type. Also, some types (e.g. `DATETIME`) are not supported yet. For more details, please refer to the [full type mapping](https://github.com/apache/beam/blob/0b430748cdd2e25edc553747ce018195e9cce888/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L112-L123). ``` ########## website/www/site/content/en/documentation/io/built-in/google-bigquery.md: ########## @@ -779,6 +779,17 @@ Starting with version 2.36.0 of the Beam SDK for Java, you can use the [BigQuery Storage Write API](https://cloud.google.com/bigquery/docs/write-api) from the BigQueryIO connector. +Also after version 2.47.0 of Beam SDK for Python, SDK supports BigQuery Storage Write API. + +{{< paragraph class="language-py" >}} +BigQuery Storage Write API for Python SDK has some limitations on supported data types. This is due to that the Python SDK uses cross-language transforms for the feature. For example, `decimal.Decimal` in Python is needed to write a `NUMERIC` type in BigQuery. Also `DATETIME` in Python is not supported yet. For more details, please refer to the [type mapping](https://github.com/apache/beam/blob/0b430748cdd2e25edc553747ce018195e9cce888/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L112-L123). +{{< /paragraph >}} + +{{< paragraph class="language-py" >}} +**Note:** If you want to run WriteToBigQuery with Storage Write API from the source code, you need to run `./gradlew :sdks:java:io:google-cloud-platform:expansion-service:build` to setup corresponding Java classes. If you want to run it from a released Beam SDK, you don't need to execute the command since the expansion service will be automatically downloaded and started. Review Comment: ```suggestion **Note:** If you want to run WriteToBigQuery with Storage Write API from the source code, you need to run `./gradlew :sdks:java:io:google-cloud-platform:expansion-service:build` to build the expansion-service jar. If you are running from a released Beam SDK, the jar will already be included. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
