tvalentyn commented on PR #38407: URL: https://github.com/apache/beam/pull/38407#issuecomment-4409342507
When backpressure happens, would that look to users as a processing lull? If so, that is less than ideal, since we probably should be throttling reading inputs. Ideally the Runner would be smart enough to not send them if it cannot consume outputs fast enough... Have you considered tracking unwritten data at the Data Channel? Something like https://github.com/apache/beam/pull/38422 perhaps (gemini assisted). I am a bit hesitant with us rewriting queue internals, but I can take a closer look if you think this approach would be better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
