+1 for this proposal. Greetings to the Apache Celeborn community~! Yuxin and I are from the Apache Flink community, and have been working on the shuffle related components for years. We are both excited about making our first contribution to the Apache Celeborn community.
Hybrid Shuffle is a new shuffle architecture that the Flink community has been working on for ~2 years. We are planning to make it the default (and eventually the only) batch shuffle in the Flink 2.0 release (end of this year). The architecture is flexible and extensible so that it can support all the capabilities of existing shuffle modes, while providing new advantages on task scheduling, resource efficiency and usability. To achieve this, we abstract storages (memory, local dist, remote storage / service) into Tiers, and hide details such as assembling records to buffers, dynamic switching between Tiers and memory management from the Tiers. We believe it is important that Flink and Celeborn can be integrated under the new architecture, in addition to the existing integration based on the shuffle-service interfaces. Looking forward to your feedback. Best, Xintong On Tue, May 28, 2024 at 8:52 PM Yuxin Tan <[email protected]> wrote: > Hi all, > > I would like to start a discussion on CIP-6 Support Flink hybrid shuffle > integration with Apache > Celeborn[1]. Celeborn provides a stable, performant, scalable remote > shuffle service. > Concurrently, Flink hybrid shuffle supports transitions between memory, > disk, and remote > storage to improve performance and job stability. This integration proposal > is to harness the > benefits from both Celeborn and hybrid shuffle simultaneously. > > Note that this proposal has two parts. > 1. The Celeborn-side changes are in CIP-6[1]. > 2. The Flink-side modifications are in FLIP-459[2]. > > Looking forward to everyone's feedback and suggestions. Thank you! > > [1] > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > Best, > Yuxin >
