Hello Xinyu, Thank you very much for your participation in the community discussion. It's been a while since your last response，and I was wondering if there has been any progress on this PR. If you still have plans to submit it to the community, that would be great. Alternatively, if you could share an overview of the design, it would be wonderful to have the opportunity for further discussion.
Thanks, Jiashu Xiong 王馨雨 <shea_w...@163.com> 于2025年6月30日周一 13:26写道： > We have a different solution to this problem. After testing, the > performance of this solution in the shuffle write phase is basically the > same as that of ESS. I will submit a PR to the community in the next few > days, and hope to discuss it together. > > > > > Regards, > > Wang Xinyu > > > > > > > > > > > > > > > > > > At 2025-06-29 22:32:31, "rexxiong" <rexxi...@apache.org> wrote: > >Thanks to Erik for the proposal. > > > >In fact, before Erik introduced this feature to the community, we had > >already discussed this idea together, and Erik's team implemented it > >internally. Later, we integrated this optimization into our production > >environment, and I must say it has significantly improved performance in > >skew scenarios. It not only enhances shuffle write efficiency notably but > >also improves cluster resource utilization, preventing overload on a few > >nodes. > > > >Additionally, there's a small issue to note: CIP-18 has already been used, > >you can use CIP-20 for this. > > > > > >Regards, > >Jiashu Xiong > > > > > >Erik fang <fme...@gmail.com> 于2025年6月27日周五 19:17写道： > > > >> Hi community, > >> > >> I'd like to start a discuss about CIP-18: Dynamically optimize shuffle > >> write parallelism > >> > >> This proposal aims to enable Celeborn to write to multiple > >> PartitionLocations for a single partition concurrently, which > significantly > >> improves skew partition performance > >> > >> Please let me know if you have any comments or questions > >> > >> link: > >> > >> > https://docs.google.com/document/d/1CqFswIOP5nR8Cy2THo8tELOwVxUf0ZP_8pDjaYj2HGc/edit?usp=sharing > >> > >> Regards, > >> Erik > >> >
