Hi all, I've written up a proposal for supporting HARD_SPLIT in Celeborn. You can find the proposal here <https://cwiki.apache.org/confluence/display/CELEBORN/CIP-12+Support+HARD_SPLIT+in+PushMergedData>. Please let me know if you have any comments or questions.Unlike PushData, Celeborn won’t actively trigger HARD_SPLIT in PushMergedData unless there are one or more partitions which have been split in the partition group of PushMergedData. This leads to several problems: Cascading HARD_SPLIT in PushMergedData will be too wasted because most partitions may not reach the HARD_SPLIT threshold.Worker pressure cannot be transferred if the partitions won’t be split.ReverveSize won’t take effect.Supporting HARD_SPLIT in PushMergedData will solve the above problems which will only split the partitions that need to be split.
Thanks, Xinyu Wang
