bvolpato opened a new pull request, #39995:
URL: https://github.com/apache/beam/pull/39995
BigQuery streaming inserts share one retry iterator across an entire bundle
and advance it even for successful requests. With `max_retries=1`, a successful
batch exhausts the retry budget for the next batch: a transient failure then
goes straight to the failed-row output. Prior successes also inflate the
initial backoff for later batches; after 20 successful batches it reaches
750–1500 seconds.
Create the retry iterator per batch, matching the documented per-group retry
limit. Partial-failure retries still preserve the failed rows and their insert
IDs. Tests cover consecutive batches, different destinations, a previously
exhausted batch, initial backoff, and zero retries.
## Testing
From `sdks/python`, with the SDK and GCP test dependencies installed:
```sh
python -m pytest apache_beam/io/gcp/bigquery_test.py -q \
-k 'successful_batches or exhausted_batch or zero_retries'
```
All 5 new cases pass. Against the upstream implementation, 4 fail and the
zero-retry compatibility case passes.
All 50 tests in the three affected streaming test classes also pass on
Python 3.12. Credential discovery was stubbed only in the test process; the
tests' existing request/client mocks remain in use:
```python
from unittest import mock
import pytest
from google.auth.credentials import AnonymousCredentials
with mock.patch('google.auth.default',
return_value=(AnonymousCredentials(), 'test-project')):
raise SystemExit(pytest.main([
'-q', '--timeout=30',
'apache_beam/io/gcp/bigquery_test.py::BigQueryStreamingInsertTransformTests',
'apache_beam/io/gcp/bigquery_test.py::BigQueryStreamingInsertsErrorHandling',
'apache_beam/io/gcp/bigquery_test.py::PipelineBasedStreamingInsertTest',
]))
```
The full module encountered an unrelated credential-lock failure in an
existing read test, also reproduced on upstream. YAPF 0.43.0, Ruff 0.15.22, and
`git diff --check` pass for the changed files.
## Downsides
The retry allowance now applies independently to each batch, so several
failing batches can make more total attempts than the previous accidental
shared limit. The configured per-batch limit remains unchanged.
------------------------
- [x] Describe the bug and include reproducible regression tests.
- [x] Update `CHANGES.md` with the behavior change.
- [ ] Apache Individual Contributor License Agreement, if required for this
contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
