Re: Trouble using dynamic allocation and shuffle service.

[Tim Chen]

Hi Philip,

I've included documentation in the Spark/Mesos doc (
http://spark.apache.org/docs/latest/running-on-mesos.html), where you can
start the MesosShuffleService with sbin/start-mesos-shuffle-service.sh
script.

The shuffle service needs to be started manually for Mesos on each slave
(one way is via Marathon with unique hostname constraint), and then you
need to enable dynamicAllocation and shuffle service flag on the driver and
it should work.

Let me know if that's not clear.

Tim

On Mon, Sep 14, 2015 at 11:36 AM, Philip Weaver 
wrote:

> Hello, I am trying to use dynamic allocation which requires the shuffle
> service. I am running Spark on mesos.
>
> Whenever I set spark.shuffle.service.enabled=true, my Spark driver fails
> with an error like this:
>
> Caused by: java.net.ConnectException: Connection refused: devspark1/
> 172.26.21.70:7337
>
> It's not clear from the documentation if the shuffle service starts
> automatically just by having it enabled, or if I need to do something else.
> There are instructions for running the shuffle service in YARN, but not
> mesos.
>
> - Philip
>
>

Re: Trouble using dynamic allocation and shuffle service.

[Philip Weaver]

Ah, I missed that, thanks!

On Mon, Sep 14, 2015 at 11:45 AM, Tim Chen  wrote:

> Hi Philip,
>
> I've included documentation in the Spark/Mesos doc (
> http://spark.apache.org/docs/latest/running-on-mesos.html), where you can
> start the MesosShuffleService with sbin/start-mesos-shuffle-service.sh
> script.
>
> The shuffle service needs to be started manually for Mesos on each slave
> (one way is via Marathon with unique hostname constraint), and then you
> need to enable dynamicAllocation and shuffle service flag on the driver and
> it should work.
>
> Let me know if that's not clear.
>
> Tim
>
> On Mon, Sep 14, 2015 at 11:36 AM, Philip Weaver 
> wrote:
>
>> Hello, I am trying to use dynamic allocation which requires the shuffle
>> service. I am running Spark on mesos.
>>
>> Whenever I set spark.shuffle.service.enabled=true, my Spark driver fails
>> with an error like this:
>>
>> Caused by: java.net.ConnectException: Connection refused: devspark1/
>> 172.26.21.70:7337
>>
>> It's not clear from the documentation if the shuffle service starts
>> automatically just by having it enabled, or if I need to do something else.
>> There are instructions for running the shuffle service in YARN, but not
>> mesos.
>>
>> - Philip
>>
>>
>