+1 (non-binding) On Tue, Jun 18, 2019 at 12:12 PM John Zhuge <john.zh...@gmail.com> wrote:
> +1 (non-binding) Great work! > > On Tue, Jun 18, 2019 at 6:22 AM Vinoo Ganesh <vgan...@palantir.com> wrote: > >> +1 (non-binding). >> >> >> >> Thanks for pushing this forward, Matt and Yifei. >> >> >> >> *From: *Felix Cheung <felixcheun...@hotmail.com> >> *Date: *Tuesday, June 18, 2019 at 00:01 >> *To: *Yinan Li <liyinan...@gmail.com>, "rb...@netflix.com" < >> rb...@netflix.com> >> *Cc: *Dongjoon Hyun <dongjoon.h...@gmail.com>, Saisai Shao < >> sai.sai.s...@gmail.com>, Imran Rashid <im...@therashids.com>, Ilan >> Filonenko <i...@cornell.edu>, bo yang <bobyan...@gmail.com>, Matt Cheah < >> mch...@palantir.com>, Spark Dev List <dev@spark.apache.org>, "Yifei >> Huang (PD)" <yif...@palantir.com>, Vinoo Ganesh <vgan...@palantir.com>, >> Imran Rashid <iras...@cloudera.com> >> *Subject: *Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API >> >> >> >> +1 >> >> >> >> Glad to see the progress in this space - it’s been more than a year since >> the original discussion and effort started. >> >> >> ------------------------------ >> >> *From:* Yinan Li <liyinan...@gmail.com> >> *Sent:* Monday, June 17, 2019 7:14:42 PM >> *To:* rb...@netflix.com >> *Cc:* Dongjoon Hyun; Saisai Shao; Imran Rashid; Ilan Filonenko; bo yang; >> Matt Cheah; Spark Dev List; Yifei Huang (PD); Vinoo Ganesh; Imran Rashid >> *Subject:* Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API >> >> >> >> +1 (non-binding) >> >> >> >> On Mon, Jun 17, 2019 at 1:58 PM Ryan Blue <rb...@netflix.com.invalid> >> wrote: >> >> +1 (non-binding) >> >> >> >> On Sun, Jun 16, 2019 at 11:11 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >> wrote: >> >> +1 >> >> >> >> Bests, >> >> Dongjoon. >> >> >> >> >> >> On Sun, Jun 16, 2019 at 9:41 PM Saisai Shao <sai.sai.s...@gmail.com> >> wrote: >> >> +1 (binding) >> >> >> >> Thanks >> >> Saisai >> >> >> >> Imran Rashid <im...@therashids.com> 于2019年6月15日周六 上午3:46写道： >> >> +1 (binding) >> >> I think this is a really important feature for spark. >> >> First, there is already a lot of interest in alternative shuffle storage >> in the community. There is already a lot of interest in alternative >> shuffle storage, from dynamic allocation in kubernetes, to even just >> improving stability in standard on-premise use of Spark. However, they're >> often stuck doing this in forks of Spark, and in ways that are not >> maintainable (because they copy-paste many spark internals) or are >> incorrect (for not correctly handling speculative execution & stage >> retries). >> >> Second, I think the specific proposal is good for finding the right >> balance between flexibility and too much complexity, to allow incremental >> improvements. A lot of work has been put into this already to try to >> figure out which pieces are essential to make alternative shuffle storage >> implementations feasible. >> >> Of course, that means it doesn't include everything imaginable; some >> things still aren't supported, and some will still choose to use the older >> ShuffleManager api to give total control over all of shuffle. But we know >> there are a reasonable set of things which can be implemented behind the >> api as the first step, and it can continue to evolve. >> >> >> >> On Fri, Jun 14, 2019 at 12:13 PM Ilan Filonenko <i...@cornell.edu> wrote: >> >> +1 (non-binding). This API is versatile and flexible enough to handle >> Bloomberg's internal use-cases. The ability for us to vary implementation >> strategies is quite appealing. It is also worth to note the minimal changes >> to Spark core in order to make it work. This is a very much needed addition >> within the Spark shuffle story. >> >> >> >> On Fri, Jun 14, 2019 at 9:59 AM bo yang <bobyan...@gmail.com> wrote: >> >> +1 This is great work, allowing plugin of different sort shuffle >> write/read implementation! Also great to see it retain the current Spark >> configuration >> (spark.shuffle.manager=org.apache.spark.shuffle.YourShuffleManagerImpl). >> >> >> >> >> >> On Thu, Jun 13, 2019 at 2:58 PM Matt Cheah <mch...@palantir.com> wrote: >> >> Hi everyone, >> >> >> >> I would like to call a vote for the SPIP for SPARK-25299 >> [issues.apache.org] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D25299&d=DwMFJg&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU&s=LS6AKX38P5DW6ffk9u5MUvRBEAlAHiA3Ud2KODpWkQU&e=>, >> which proposes to introduce a pluggable storage API for temporary shuffle >> data. >> >> >> >> You may find the SPIP document here [docs.google.com] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1d6egnL6WHOwWZe8MWv3m8n4PToNacdx7n-5F0iMSWwhCQ_edit&d=DwMFJg&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU&s=rCSgQGD6L4of4oa0QxiTJ8IPaVdGlZVarhA4-QvO80Q&e=> >> . >> >> >> >> The discussion thread for the SPIP was conducted here [lists.apache.org] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_2fe82b6b86daadb1d2edaef66a2d1c4dd2f45449656098ee38c50079-40-253Cdev.spark.apache.org-253E&d=DwMFJg&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU&s=kSJizQH7v4OHG6D7aVsLA-m0ApZxOa24CzHZv1EzLxg&e=> >> . >> >> >> >> Please vote on whether or not this proposal is agreeable to you. >> >> >> >> Thanks! >> >> >> >> -Matt Cheah >> >> >> >> >> -- >> >> Ryan Blue >> >> Software Engineer >> >> Netflix >> >> > > -- > John > -- -Dhruve Ashar
