Is this what you are looking for
1. Build Spark with the YARN profile
http://spark.apache.org/docs/1.2.0/building-spark.html. Skip this step
if you are using a pre-packaged distribution.
2. Locate the spark-version-yarn-shuffle.jar. This should be under
I've read that this is supposed to be a rather significant optimization to
the shuffle system in 1.1.0 but I'm not seeing much documentation on
enabling this in Yarn. I see github classes for it in 1.2.0 and a property
spark.shuffle.service.enabled in the spark-defaults.conf.
The code mentions
