Thanks for writing this up. Usually when people talk about push-based shuffle, they are motivating it primarily to reduce the latency of short queries, by pipelining the map phase, shuffle phase, and the reduce phase (which this design isn't going to address). It's interesting you are targeting throughput by optimizing for random reads instead.
My main questions are ... 1. This is designing for HDDs. But SSD prices have gone lower than HDDs this year, so most new data center storage will be using SSDs from now on. Are we introducing a lot of complexity to address a problem that only exists with legacy that will be phased out soon? 2. Is there a simpler way to address this? E.g. you can simply merge map outputs for each node locally, without involving any type of push. It seems to me you'd address the same issues you have, with the same limitations (of memory buffer limiting the number of concurrent streams you can write to). On Tue, Jan 21, 2020 at 6:13 PM, mshen < ms...@apache.org > wrote: > > > > I'd like to start a discussion on enabling push-based shuffle in Spark. > This is meant to address issues with existing shuffle inefficiency in a > large-scale Spark compute infra deployment. > Facebook's previous talks on SOS shuffle > < https:/ / databricks. com/ session/ sos-optimizing-shuffle-i-o ( > https://databricks.com/session/sos-optimizing-shuffle-i-o ) > and Cosco > shuffle service > < https:/ / databricks. com/ session/ > cosco-an-efficient-facebook-scale-shuffle-service > ( > https://databricks.com/session/cosco-an-efficient-facebook-scale-shuffle-service > ) > are solutions dealing with a similar problem. > Note that this is somewhat orthogonal to the work in SPARK-25299 > < https:/ / issues. apache. org/ jira/ browse/ SPARK-25299 ( > https://issues.apache.org/jira/browse/SPARK-25299 ) > , which is to use > remote storage to store shuffle data. > More details of our proposed design is in SPARK-30602 > < https:/ / issues. apache. org/ jira/ browse/ SPARK-30602 ( > https://issues.apache.org/jira/browse/SPARK-30602 ) > , with SPIP attached. > Would appreciate comments and discussions from the community. > > > > ----- > Min Shen > Staff Software Engineer > LinkedIn > -- > Sent from: http:/ / apache-spark-developers-list. 1001551. n3. nabble. com/ > ( http://apache-spark-developers-list.1001551.n3.nabble.com/ ) > > > > --------------------------------------------------------------------- To > unsubscribe e-mail: dev-unsubscribe@ spark. apache. org ( > dev-unsubscr...@spark.apache.org ) > > >
