Trouble using dynamic allocation and shuffle service.

Hello, I am trying to use dynamic allocation which requires the shuffle service. I am running Spark on mesos. Whenever I set spark.shuffle.service.enabled=true, my Spark driver fails with an error like this: Caused by: java.net.ConnectException: Connection refused: devspark1/ 172.26.21.70:7337

Re: Trouble using dynamic allocation and shuffle service.

Hi Philip, I've included documentation in the Spark/Mesos doc ( http://spark.apache.org/docs/latest/running-on-mesos.html), where you can start the MesosShuffleService with sbin/start-mesos-shuffle-service.sh script. The shuffle service needs to be started manually for Mesos on each slave (one

Re: Trouble using dynamic allocation and shuffle service.

Ah, I missed that, thanks! On Mon, Sep 14, 2015 at 11:45 AM, Tim Chen wrote: > Hi Philip, > > I've included documentation in the Spark/Mesos doc ( > http://spark.apache.org/docs/latest/running-on-mesos.html), where you can > start the MesosShuffleService with