mohamedawnallah commented on code in PR #37108: URL: https://github.com/apache/beam/pull/37108#discussion_r2618977838
########## website/www/site/content/en/documentation/io/built-in/google-bigquery.md: ########## @@ -659,6 +659,11 @@ runtime. The sharding behavior depends on the runners. You must use `triggering_frequency` to specify a triggering frequency for initiating load jobs. Be careful about setting the frequency such that your pipeline doesn't exceed the BigQuery load job [quota limit](https://cloud.google.com/bigquery/quotas#load_jobs). + +> **Note:** When using file load-based BigQuery writes with dynamic destinations and a non-zero +> `triggering_frequency`, temporary tables may be created repeatedly and loads +> are not finalized into destination tables. This is a known limitation (see BEAM-9917). Review Comment: > As a next step, I can add a failing unit test that captures the current behavior with dynamic destinations and triggering_frequency, or help with investigation into the finalization logic. Please let me know which direction would be preferred. It would be great if we could have a test that captures the bug first, then we can iterate on the solution ########## website/www/site/content/en/documentation/io/built-in/google-bigquery.md: ########## @@ -659,6 +659,11 @@ runtime. The sharding behavior depends on the runners. You must use `triggering_frequency` to specify a triggering frequency for initiating load jobs. Be careful about setting the frequency such that your pipeline doesn't exceed the BigQuery load job [quota limit](https://cloud.google.com/bigquery/quotas#load_jobs). + +> **Note:** When using file load-based BigQuery writes with dynamic destinations and a non-zero +> `triggering_frequency`, temporary tables may be created repeatedly and loads +> are not finalized into destination tables. This is a known limitation (see BEAM-9917). Review Comment: > As a next step, I can add a failing unit test that captures the current behavior with dynamic destinations and triggering_frequency, or help with investigation into the finalization logic. Please let me know which direction would be preferred. It would be great if we could have a reproducible test that captures the bug first, then we can iterate on the solution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
