Hi Yuxin, Thanks for driving this CIP about integration with Hybrid Shuffle. I have some comments on this CIP:
1. Could you describe in detail what functions the relevant components mentioned in Proposed Changes, including CelebornProducerAgent, CelebornConsumerAgent, CelebornMasterAgent, etc., support? In the design document, these components are only mentioned and no any details of changes. 2. Can you briefly introduce how to guarantee compatibility with Celeborn’s existing features such as partition splitting? IMO, the compatibility introduction should be mentioned in Proposed Changes to help community developers understand. 3. There are no changes on public interfaces. Is there any public configuration of integration with Hybrid Shuffle and Flink client? 4. The server side must store Segment information for each subpartition. How does the server side guarantee the accuracy and recoverability of Segment information? 5. Should Celeborn wait until FLIP-459 is released before releasing this integration? Which Flink version will release FLIP-459? Regards, Nicholas Jiang On 2024/05/28 12:51:32 Yuxin Tan wrote: > Hi all, > > I would like to start a discussion on CIP-6 Support Flink hybrid shuffle > integration with Apache > Celeborn[1]. Celeborn provides a stable, performant, scalable remote > shuffle service. > Concurrently, Flink hybrid shuffle supports transitions between memory, > disk, and remote > storage to improve performance and job stability. This integration proposal > is to harness the > benefits from both Celeborn and hybrid shuffle simultaneously. > > Note that this proposal has two parts. > 1. The Celeborn-side changes are in CIP-6[1]. > 2. The Flink-side modifications are in FLIP-459[2]. > > Looking forward to everyone's feedback and suggestions. Thank you! > > [1] > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > Best, > Yuxin >
