Thanks Xinyu for the proposal! Glad to see PushMergeData will support
splitting finally :)
The design LGTM overall, still I have some comments:
1. We can also support soft split as well
2. Typically there are more than one disks, so it's better to only splits
partitions
belonging to full disks instead of splitting all partitions
3. We can group partitions by disks to increase efficiency when checking
whether disk is full for each partition
4. For FILE_ALREADY_CLOSED I think we shouldn't decrease remainReviveTimes
either, it's not an exception or error
Regards,
Keyong Zhou
Fu Chen <[email protected]> 于2024年9月21日周六 18:03写道：
> Thanks Xinyu for the proposal. Adding HARD_SPLIT support for
> PushMergeData is valuable for production. We've encountered issues
> with small disk nodes getting overloaded in heterogeneous clusters.
>
> I had a discussion with @rexxiong, the current implementation requires
> introducing PUSH_MERGED_DATA_RESPONSE, which increases the complexity
> of modifications. We could consider reusing the RpcResponse
>
> Thanks,
> Fu Chen
>
> 王馨雨 <[email protected]> 于2024年9月20日周五 10:35写道：
> >
> > Hi all,
> >
> > I've written up a proposal for supporting HARD_SPLIT in Celeborn. You
> can find
> > the proposal here
> > <
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-12+Support+HARD_SPLIT+in+PushMergedData
> >.
> > Please let me know if you have any comments or questions.Unlike
> PushData, Celeborn won’t actively trigger HARD_SPLIT in PushMergedData
> unless there are one or more partitions which have been split in the
> partition group of PushMergedData.
> > This leads to several problems:
> > Cascading HARD_SPLIT in PushMergedData will be too wasted because most
> partitions may not reach the HARD_SPLIT threshold.Worker pressure cannot be
> transferred if the partitions won’t be split.ReverveSize won’t take
> effect.Supporting HARD_SPLIT in PushMergedData will solve the above
> problems which will only split the partitions that need to be split.
> >
> > Thanks,
> > Xinyu Wang
> >
>
