tvalentyn commented on code in PR #40598:
URL: https://github.com/apache/beam/pull/40598#discussion_r4285908904
##########
sdks/python/apache_beam/io/components/adaptive_throttler.py:
##########
@@ -155,16 +165,18 @@ def __init__(
Args:
window_ms: int, length of history to consider, in ms, to set
- throttling.
+ throttling. Should be much larger than throttle_delay_secs and any
+ retry backoff between attempts, otherwise failures expire from the
+ history before they can influence throttling.
Review Comment:
I think the spirit of adaptive throttling is the following: we should be
probabilistically rejecting the requests in the SDK, not just delaying the
requests, and fall back into the exponential back off mechanism, yet provide a
signal to dataflow that SDK is doing useful work (waiting in exponential back
off, but it is not stuck, so don't upscale).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
