Yuxin Tan created FLINK-33742: --------------------------------- Summary: Hybrid Shuffle should work well with Adaptive Query Execution Key: FLINK-33742 URL: https://issues.apache.org/jira/browse/FLINK-33742 Project: Flink Issue Type: Improvement Components: Runtime / Coordination, Runtime / Network Affects Versions: 1.19.0 Reporter: Yuxin Tan
At present, Hybrid Shuffle and Adaptive Query Execution (AQE), which includes features such as Dynamic Partition Pruning (DPP), Runtime Filter, and Adaptive Batch Scheduler, are not fully compatible. While they can be used concurrently at the same time, the activation of AQE inhibits the key capability of Hybrid Shuffle to perform simultaneous reading and writing. This limitation arises because AQE dictates that downstream tasks may only initiate once upstream tasks have finished, a requirement that is inconsistent with the simultaneous read-write process facilitated by Hybrid Shuffle. To harness the full potential of Hybrid Shuffle and AQE, it is essential to refine their integration. By doing so, we can capitalize on each feature's distinct advantages and enhance overall system performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)