prodriguezdefino opened a new pull request, #40606: URL: https://github.com/apache/beam/pull/40606
When a bundle's input queue is full, `BeamFnDataGrpcMultiplexer` blocks in `onNext`. gRPC runs `onNext` and the outbound `onReady` callback on the same executor, so a bundle thread that waits in `DirectStreamObserver` for `onReady` only continues when its wait times out (up to 60 s). The runner logs "Ignoring data element byte limit … to avoid stuckness" and the job crawls or stalls. We hit this on Dataflow with a Java step fused between two steps of another SDK. The inbound data stream now uses manual gRPC flow control, so `onNext` never blocks. A message that does not fit in the bundle queue waits in a buffer, and the stream asks for more messages as bundles finish them. While output waits on flow control, it asks for one more message every 500 ms, up to 100 for each active bundle. `DirectStreamObserver` checks `isReady()` before each message. Related to #20028. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
