Hi Philip, I've included documentation in the Spark/Mesos doc ( http://spark.apache.org/docs/latest/running-on-mesos.html), where you can start the MesosShuffleService with sbin/start-mesos-shuffle-service.sh script.
The shuffle service needs to be started manually for Mesos on each slave (one way is via Marathon with unique hostname constraint), and then you need to enable dynamicAllocation and shuffle service flag on the driver and it should work. Let me know if that's not clear. Tim On Mon, Sep 14, 2015 at 11:36 AM, Philip Weaver <philip.wea...@gmail.com> wrote: > Hello, I am trying to use dynamic allocation which requires the shuffle > service. I am running Spark on mesos. > > Whenever I set spark.shuffle.service.enabled=true, my Spark driver fails > with an error like this: > > Caused by: java.net.ConnectException: Connection refused: devspark1/ > 172.26.21.70:7337 > > It's not clear from the documentation if the shuffle service starts > automatically just by having it enabled, or if I need to do something else. > There are instructions for running the shuffle service in YARN, but not > mesos. > > - Philip > >
