gariepyalex opened a new issue, #35356:
URL: https://github.com/apache/beam/issues/35356
### What happened?
I'm using Dataflow 2.60 from Python and I have the following BigQuery sink
configuration:
```python
_bigquery = record | "write to BigQuery" >> beam.io.WriteToBigQuery(
output_table,
schema=schema,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
create_disposition=beam.io.BigQueryDisposition.CREATE_NEVER,
insert_retry_strategy=beam.io.gcp.bigquery_tools.RetryStrategy.RETRY_ON_TRANSIENT_ERROR,
)
```
After a schema change in the table, the write has start to fail silently,
with a warning log level. *I would expect an error log level to be used
instead*. This is what the code also suggests:
https://github.com/apache/beam/blob/v2.60.0/sdks/python/apache_beam/io/gcp/bigquery.py#L1714
The same code is present on both 2.60 and `master`.
My logs on Dataflow show:
```
{
"insertId": "6514355326739548575:173475:0:71347504",
"jsonPayload": {
"worker": "product-nomic-finetune-re-06171143-sbar-harness-l8bh",
"portability_worker_id": "sdk-1-0_sibling_12",
"message": "There were errors inserting to BigQuery. Will not retry.
Errors were [{'index': 0, 'errors': [{'reason': 'invalid', 'location':
'COLUMN_NAME', 'debugInfo': '', 'message': 'no such field: COLUMN_NAME.'}]}]",
"work": "100622",
"step": "write to
BigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn)",
"logger":
"/usr/local/lib/python3.10/dist-packages/apache_beam/io/gcp/bigquery.py:1719",
"thread": "Thread-19",
"instruction": "process_bundle-100622-243345",
"job": "2025-06-17_11_43_07-417089181064889364"
},
"resource": {
"type": "dataflow_step",
"labels": {
"job_id": "2025-06-17_11_43_07-417089181064889364",
"step_id": "write to
BigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn)",
"region": "us-east1",
}
},
"timestamp": "2025-06-17T18:59:59.449218511Z",
"severity": "WARNING",
"logName":
"projects/shopify-ml-production/logs/dataflow.googleapis.com%2Fworker",
"receiveTimestamp": "2025-06-17T19:00:04.163854367Z"
}
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
