Gyula Fora created FLINK-31215: ---------------------------------- Summary: Backpropagate processing rate limits from non-scalable bottlenecks to upstream operators Key: FLINK-31215 URL: https://issues.apache.org/jira/browse/FLINK-31215 Project: Flink Issue Type: New Feature Components: Autoscaler, Kubernetes Operator Reporter: Gyula Fora
The current algorithm scales operators based on input data rates by propagating it forward through the graph. However there are cases where a certain operators processing capacity is limited either because it has a set maxParallelism or the users excludes it from scaling (or otherwise the capacity doesnt increase with scaling). In these cases it doesn't make sense to scale upstream operators to the target data rate if the job is going to be bottlenecked by a downstream operator. But instead we should backpropagate the limit based on the non-scalable bottleneck. -- This message was sent by Atlassian Jira (v8.20.10#820010)