+1 (non-binding). This API is versatile and flexible enough to handle Bloomberg's internal use-cases. The ability for us to vary implementation strategies is quite appealing. It is also worth to note the minimal changes to Spark core in order to make it work. This is a very much needed addition within the Spark shuffle story.
On Fri, Jun 14, 2019 at 9:59 AM bo yang <bobyan...@gmail.com> wrote: > +1 This is great work, allowing plugin of different sort shuffle > write/read implementation! Also great to see it retain the current Spark > configuration > (spark.shuffle.manager=org.apache.spark.shuffle.YourShuffleManagerImpl). > > > On Thu, Jun 13, 2019 at 2:58 PM Matt Cheah <mch...@palantir.com> wrote: > >> Hi everyone, >> >> >> >> I would like to call a vote for the SPIP for SPARK-25299 >> <https://issues.apache.org/jira/browse/SPARK-25299>, which proposes to >> introduce a pluggable storage API for temporary shuffle data. >> >> >> >> You may find the SPIP document here >> <https://docs.google.com/document/d/1d6egnL6WHOwWZe8MWv3m8n4PToNacdx7n_0iMSWwhCQ/edit> >> . >> >> >> >> The discussion thread for the SPIP was conducted here >> <https://lists.apache.org/thread.html/2fe82b6b86daadb1d2edaef66a2d1c4dd2f45449656098ee38c50079@%3Cdev.spark.apache.org%3E> >> . >> >> >> >> Please vote on whether or not this proposal is agreeable to you. >> >> >> >> Thanks! >> >> >> >> -Matt Cheah >> >
