[ https://issues.apache.org/jira/browse/FLINK-31655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711909#comment-17711909 ]
Piotr Nowojski commented on FLINK-31655: ---------------------------------------- I think there are ways to address problems with [~akalash] proposal, without adding too much overhead (also checking {{getWritingThreadTotalNumberOfSentBytes}} for the channel that we haven't selected?). I think both of your proposed solutions would be quite costly [~tartarus], adding extra synchronisations that we don't necessarily need with some clever heuristic. I agree with [~pltbkd], we would probably need a FLIP, as this will be changing public APIs (either changing behaviour of an existing {{rebalance()}}, or adding a new one like {{loadRebalance()}}). > Adaptive Channel selection for partitioner > ------------------------------------------ > > Key: FLINK-31655 > URL: https://issues.apache.org/jira/browse/FLINK-31655 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task > Reporter: tartarus > Assignee: tartarus > Priority: Major > > In Flink, if the upstream and downstream operator parallelism is not the > same, then by default the RebalancePartitioner will be used to select the > target channel. > In our company, users often use flink to access redis, hbase or other rpc > services, If some of the Operators are slow to return requests (for external > service reasons), then because Rebalance/Rescale are Round-Robin the Channel > selection policy, so the job is easy to backpressure. > Because the Rebalance/Rescale policy does not care which subtask the data is > sent to downstream, so we expect Rebalance/Rescale to refer to the processing > power of the downstream subtask when choosing a Channel. > Send more data to the free subtask, this ensures the best possible throughput > of job! > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)