+1 for the FLIP It is useful to adaptively optimize logical execution plans(stream operators and stream edges) for batch jobs.
One question: The FLIP already proposed to update the REST API & Web UI to show operators that are not yet converted to job vertices. However, I think it would be better if Flink can display these operators as part of the graph, allowing users to have an overview of the processing logic graph at early stages of the job execution. This change would also involve the public interface, so instead of postponing it to a later FLIP, I prefer to have a design for it in this FLIP. WDYT? Thanks, Zhu Junrui Lee <jrlee....@gmail.com> 于2024年7月11日周四 11:27写道： > Hi devs, > > Xia Sun, Lei Yang, and I would like to initiate a discussion about > FLIP-469: Supports Adaptive Optimization of StreamGraph. > > This FLIP is the second in the series on adaptive optimization of > StreamGraph and follows up on FLIP-468 [1]. As we proposed in FLIP-468 to > enable the scheduler to recognize and access the StreamGraph, in this FLIP, > we will propose a mechanism for adaptive optimization of StreamGraph. It > allows the scheduler to dynamically adjust the logical execution plan at > runtime. This mechanism is the base of adaptive optimization strategies, > such as adaptive broadcast join and skewed join optimization. > > Unlike the traditional execution of jobs based on a static StreamGraph, > this mechanism will progressively determine StreamGraph during runtime. The > determined StreamGraph will be transformed into a specific JobGraph, while > the indeterminate part will allow Flink to flexibly adjust according to > real-time job status and actual input conditions. > > Note that this FLIP focuses on the introduction of the mechanism and does > not introduce any actual optimization strategies; these will be discussed > in subsequent FLIPs. > > For more details, please refer to FLIP-469 [2]. We look forward to your > feedback. > > Best, > > Xia Sun, Lei Yang and Junrui Lee > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-469%3A+Supports+Adaptive+Optimization+of+StreamGraph >
