Abacn commented on PR #31924: URL: https://github.com/apache/beam/pull/31924#issuecomment-2236670761
before - agressively scale up until get completely stuck, then no longer scale up because average CPU usage below 5% <img width="861" alt="image" src="https://github.com/user-attachments/assets/e768b2c9-b5a0-43a7-9a0c-348e98e20043";> after - multiple scale down event <img width="866" alt="image" src="https://github.com/user-attachments/assets/fd84d496-b737-4b2b-b883-d4606541263a";> though there are some caveat - once throttling signal relieved, autoscaler tends to scale up again, causing bouncing between throttling-relieved - the last batch seems stuck to flush indefinitely. This is a known issue for BigtableIO: #29548 introduced attemptedTimeout config to force the future expire in finite time to mitigate the issue, but the underlying issue always present -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
