Github user yucai commented on the issue:
https://github.com/apache/spark/pull/19788
@cloud-fan @gatorsmile I am trying the new method as suggested and I have a
question.
If we make it **purely server-side** optimization, for external shuffle
service, it has no idea how shuffle data is compressed (concatenatable?) or
serialized (relocatable?), how does it decide if it can merge the contiguous
partition or not?
One possible solution is to read all contiguous partition in one shot and
then send the data one by one, how do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
