bvolpato opened a new pull request, #40001:
URL: https://github.com/apache/beam/pull/40001
Python BigQuery `FILE_LOADS` can silently omit rows when a dynamic
destination returns different supported spellings of the same table. With
`project='project'`, both `project:dataset.table` and `dataset.table` identify
one destination table, but file grouping retains the original strings. Each
group starts at partition zero, while load-job IDs previously hashed the
resolved table reference. The resulting ID collision makes BigQuery treat the
second group's files as an already-submitted job.
For two input rows routed to these two spellings, a public `WriteToBigQuery`
pipeline finishes successfully with only one row written. Using the same
spelling for both rows writes both. The colon/dot qualified aliases reproduce
the same defect.
Hash the same destination identity used by file grouping. Separate groups
now receive separate load jobs and temporary tables, while retries of a group
retain their deterministic IDs. Original schema and additional-parameter
callback arguments are preserved. Copy-job bookkeeping already resolves the
physical table, so only the first copy uses `WRITE_TRUNCATE` when requested.
## Reproduction and testing
From `sdks/python`, with SDK and GCP test dependencies installed:
```sh
python -m pytest apache_beam/io/gcp/bigquery_file_loads_test.py -q \
-k equivalent_destinations_keep_distinct_load_jobs
```
Both new cases fail against upstream and pass with the fix. They exercise
actual load/copy DoFns and `BigQueryWrapper` with a Jobs endpoint that
deduplicates job IDs, covering append/truncate modes, every source file
reaching a copy job, replay stability, canonical-ID compatibility, and callback
arguments.
The non-integration FILE_LOADS suite passes: 38 tests, with 6 integration
tests deselected and credential discovery stubbed only in the test process. A
separate public-transform reproduction using `BundleBasedDirectRunner`, local
temporary files, and a fake Jobs endpoint passes all 3 cases; upstream loses a
row in both alias cases while the identical-spelling control passes. No live
BigQuery service was used.
YAPF 0.43.0, Ruff 0.15.22, and `git diff --check` pass.
## Downsides
Noncanonical destination spellings now receive different load-job IDs.
Replaying unfinished loads across an in-place SDK update can therefore resubmit
a previously completed load under a new ID. Canonical `project:dataset.table`
destinations retain their existing IDs, and retries within either version
remain stable.
------------------------
- [x] Describe the bug and include reproducible regression tests.
- [x] Update `CHANGES.md` with the behavior change.
- [ ] Apache Individual Contributor License Agreement, if required for this
contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
