Hi Yuxin and Xintong, Really excited to see Flink and Celeborn communities collaborate more on shuffle component! I believe this will inspire more for both sides :)
+1 for this proposal, looking forward to see this feature to make progress. Also I'm very interested in integrating Flink Hybrid Shuffle with Celeborn's Reduce Partition as mentioned in the doc in the future, which I believe will benefit more for very large shuffle operators :) Regards, Keyong Zhou Nicholas Jiang <[email protected]> 于2024年6月6日周四 13:25写道： > Hi Yuxin, > > Thanks for driving this CIP about integration with Hybrid Shuffle. I have > some comments on this CIP: > > 1. Could you describe in detail what functions the relevant components > mentioned in Proposed Changes, including CelebornProducerAgent, > CelebornConsumerAgent, CelebornMasterAgent, etc., support? In the design > document, these components are only mentioned and no any details of changes. > > 2. Can you briefly introduce how to guarantee compatibility with > Celeborn’s existing features such as partition splitting? IMO, the > compatibility introduction should be mentioned in Proposed Changes to help > community developers understand. > > 3. There are no changes on public interfaces. Is there any public > configuration of integration with Hybrid Shuffle and Flink client? > > 4. The server side must store Segment information for each subpartition. > How does the server side guarantee the accuracy and recoverability of > Segment information? > > 5. Should Celeborn wait until FLIP-459 is released before releasing this > integration? Which Flink version will release FLIP-459? > > Regards, > Nicholas Jiang > > On 2024/05/28 12:51:32 Yuxin Tan wrote: > > Hi all, > > > > I would like to start a discussion on CIP-6 Support Flink hybrid shuffle > > integration with Apache > > Celeborn[1]. Celeborn provides a stable, performant, scalable remote > > shuffle service. > > Concurrently, Flink hybrid shuffle supports transitions between memory, > > disk, and remote > > storage to improve performance and job stability. This integration > proposal > > is to harness the > > benefits from both Celeborn and hybrid shuffle simultaneously. > > > > Note that this proposal has two parts. > > 1. The Celeborn-side changes are in CIP-6[1]. > > 2. The Flink-side modifications are in FLIP-459[2]. > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > [1] > > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > > > Best, > > Yuxin > > >
