Re: Push-based shuffle SPIP

Hi, I have updated the results of running a large-shuffle application and TPCH queries in this document . The new test results which are linked from the document are here

Re: Push-based shuffle SPIP

The linked doc with detailed information of the branch does not seem to be shareable publicly. We have created a copy of the doc which should be publicly accessible. https://docs.google.com/document/d/1Q5m7YAp0HyG_TNFL4p_bjQgzzw33ik5i49Vr86UNZgg/edit?usp=sharing - Min Shen Staff Software

Re: Push-based shuffle SPIP

Hi, Thanks for sending out the proposal Min ! For the SPIP requirements, I am willing to act as the shepherd for this proposal. The jira + paper + proposal provides the high level design and implementation details. The vldb paper discusses the performance gains in detail for the inhouse

Push-based shuffle SPIP

We raised this SPIP ticket in https://issues.apache.org/jira/browse/SPARK-30602 earlier this year. Since then, we have progressed in multiple fronts, including: * Our work is published in VLDB 2020. The final version of the paper is attached in the SPIP ticket. * We have further enhanced and