damccorm commented on code in PR #41255:
URL: https://github.com/apache/beam/pull/41255#discussion_r4559984115
##########
runners/google-cloud-dataflow-java/build.gradle:
##########
@@ -556,7 +556,7 @@ task validatesRunner {
}
def validatesRunnerStreamingConfig = [
- pipelineOptions: legacyPipelineOptions + ['--streaming'],
+ pipelineOptions: legacyPipelineOptions + ['--streaming',
'--testTimeoutSeconds=1800'],
Review Comment:
Yes — this started after [PR
#40292](https://github.com/apache/beam/pull/40292) (`43bd833c88a`, merged Sep
30), [PR #40303](https://github.com/apache/beam/pull/40303) (`248548d64f0`,
merged Oct 22), and [PR #40952](https://github.com/apache/beam/pull/40952)
(`3fd91ba05b8`, merged Nov 2):
1. **2x higher streaming job concurrency than pre-#40292**: Before #40292,
each of the 12 Gradle worker JVMs (`--max-workers=12`) ran 1 `@Test` method at
a time (**12 concurrent streaming jobs**). #40292 introduced
`BeamParallelJunit4Runner` (`parallelThreads = 6`), #40303 merged batch
ValidatesRunner tests into shared Dataflow jobs (`testBatching = true` for
batch only, leaving streaming tests at 1 standalone Dataflow job per test), and
#40952 reduced streaming `parallelThreads` from `6` to `2` — which still runs
up to **`12 Gradle worker JVMs * 2 parallelThreads = 24` concurrent streaming
Dataflow jobs** in `us-central1` (plus `PostCommit Java ValidatesRunner
Dataflow V2` triggering at the same `06:30 UTC` cron schedule).
2. **Tolerating transient stockout retries in streaming (#40952)**: #40952
also updated `TestDataflowRunner` in streaming mode not to fail/cancel
immediately on transient service errors (`"The service will retry"`).
3. **What happened in the two timed-out Dataflow jobs**:
- In [run
43281087492](https://github.com/apache/beam/actions/runs/43281087492) (job
`2026-11-12_11_56_13-3714750566918859942`), the job was created at `19:56:13Z`
and sat in `JOB_STATE_QUEUED` for **12m40s** waiting for regional capacity
before Dataflow started worker pool setup at `20:08:53Z`; the worker began
receiving work at `20:10:55Z`, leaving only 19s before `testTimeoutSeconds =
900` (measured from job submission) cancelled it at `20:11:14Z`.
- In [run
43169620883](https://github.com/apache/beam/actions/runs/43169620883) (job
`2026-11-11_23_51_12-457120166125553798`), the job was created at `07:51:13Z`,
finished processing all elements and started auto-draining (`Stopping worker
pool...`) at `07:55:03Z` (3m50s after creation), but GCE worker pool teardown
under concurrent load took 12 minutes (`07:55:03Z` -> `08:07:06Z`), reaching
`JOB_STATE_DONE` at `08:07:38Z` (`16m25s` total, 85s after `testTimeoutSeconds
= 900` cancelled it at `08:06:14Z`).
Let me know if you prefer keeping `testTimeoutSeconds = 1800` (or also
setting streaming `parallelThreads` from `'2'` to `'1'` in
`configureValidatesRunnerParallelism` to restore the pre-#40292 cap of 12
concurrent streaming jobs across the 12 Gradle worker JVMs)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
