bvolpato commented on code in PR #39938:
URL: https://github.com/apache/beam/pull/39938#discussion_r3903979121
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -4227,10 +4236,15 @@ private <DestinationT> WriteResult continueExpandTyped(
batchLoads.setMaxFilesPerPartition(getMaxFilesPerPartition());
batchLoads.setMaxBytesPerPartition(getMaxBytesPerPartition());
- // When running in streaming (unbounded mode) we want to retry failed
load jobs
- // indefinitely. Failing the bundle is expensive, so we set a fairly
high limit on retries.
- if (IsBounded.UNBOUNDED.equals(input.isBounded())) {
- batchLoads.setMaxRetryJobs(getMaxRetryJobs());
+ // an explicit withMaxRetryJobs applies to batch and streaming alike.
left unset, streaming
+ // retries a failed load job far more often than batch does: failing
the bundle in streaming
+ // is expensive, so we would rather keep retrying the job than hand
the work back to the
+ // runner. batch leaves BatchLoads on its own lower default
+ Integer maxRetryJobs = getMaxRetryJobs();
Review Comment:
could this preserve old bounded default when rebuilding a write from a
legacy config row? older versions always serialized `max_retry_jobs=1000`, even
when `withMaxRetryJobs()` was never called. `fromConfigRow()` restores that
non-null value, so this branch appears to treat it as explicit and could change
bounded `FILE_LOADS` retries from 3 to 1000.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
