Re: Spark 3.2.0 first RC next week

2021-08-10 Thread Min Shen
Hi Gengliang, SPARK-36378 (Switch to using RPCResponse to communicate common block push failures to the client) should be another one. This introduces a slight protocol change to push-based shuffle to improve code robustness and performance, and is almost ready to be committed. Because of the

Re: Apache Spark 3.2 Expectation

2021-06-16 Thread Min Shen
.apache.org/jira/browse/SPARK-27790; > >> >- Support Timestamp without time zone data type > >> >https://issues.apache.org/jira/browse/SPARK-35662; > >> >- Aggregate (Min/Max/Count) push down for Parquet > >> >https://issues.

Re: Enabling push-based shuffle in Spark

2020-01-27 Thread Min Shen
ark > shuffle system's scalability and efficiency, making Spark an even > better > compute engine. > Would like to explore how we can leverage the shuffle plugin API to > make > this d