Thanks Yuepeng for driving this FLIP, it could avoid bottlenecks when part of downstream subtask is under pressure.
I prefer the Candidate design 1 that mentioned in FLIP for 2 reasons: 1. The main design does not support fine grain settings for flink sql, but nowadays Flink SQL is becoming increasingly popular. 2. As you mentioned in FLIP, the main design is complex. Not sure if it is possible to introduce the adaptive partitioner on job level in this FLIP first, and improve it for both jar and sql jobs if the fine grained is needed in the future. WDYT? > taskmanager.network.adaptive-partitioner.enable Also, one small tip: most enabled configuration option names end with "enabled". Best, Rui On Wed, Jan 7, 2026 at 9:34 AM Yuepeng Pan <[email protected]> wrote: > Bumping this thread. Thanks! > > Best regards, > Yuepeng Pan > > > Pan Yuepeng <[email protected]> 于2025年12月15日周一 12:40写道： > > > Hi devs, > > > > I re-sorted out and supplemented the 'FLIP-339[1] Support Adaptive > > Partition Selection for StreamPartitioner' based on Flink JIRA[2]. > > > > Flink offers multiple partition strategies, some of which bind data to > > downstream subtasks, while others do not (e.g., shuffle, rescale, > > rebalance). > > For [Data not bound to subtasks] scenarios, overloaded sub-task-nodes may > > slow down the processing of Flink jobs, leading to backpressure and data > > lag. Dynamically adjusting the partition of data to subtasks based on the > > processing load of downstream operators helps achieve a peak-shaving and > > valley-filling effect, thereby striving to maintain the throughput of > Flink > > jobs. > > > > The raw discussions could be found in the Flink JIRA[2]. > > I really appreciate developers involved in the discussion for the > valuable > > help and suggestions in advance. > > > > Please refer to the FLIP[1] wiki for more details about the proposed > > design and implementation. > > > > Welcome any feedback and opinions on this proposal. > > > > [1] https://cwiki.apache.org/confluence/x/nYyzDw > > [2] https://issues.apache.org/jira/browse/FLINK-31655 > > > > Best regards, > > Yuepeng Pan > > >
