Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread jleaniz
You've got to start the shuffle service on all your workers. There's a script for that in the 'sbin' directory. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-dynamic-allocation-and-shuffle-service-in-Standalone-Mode-tp26430p26434.html Sent from the

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Yuval Itzchakov
gt;> >> >> >> >> *From: *Yuval Itzchakov <yuva...@gmail.com> >> *Sent: *Tuesday, March 8, 2016 2:17 PM >> *To: *Silvio Fiorito <silvio.fior...@granturing.com>; >> user@spark.apache.org >> *Subject: *Re: Using dynamic allocation and shuffle se

RE: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Silvio Fiorito
<mailto:user@spark.apache.org> Subject: Re: Using dynamic allocation and shuffle service in Standalone Mode Actually, I assumed that setting the flag in the spark job would turn on the shuffle service in the workers. I now understand that assumption was wrong. Is there any way to set the flag via t

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Yuval Itzchakov
Actually, I assumed that setting the flag in the spark job would turn on the shuffle service in the workers. I now understand that assumption was wrong. Is there any way to set the flag via the driver? Or must I manually set it via spark-env.sh on each worker? On Tue, Mar 8, 2016, 20:14 Silvio

Re: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Suniti Singh
Please check the document for the configuration - http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup On Tue, Mar 8, 2016 at 10:14 AM, Silvio Fiorito < silvio.fior...@granturing.com> wrote: > You’ve started the external shuffle service on all worker nodes, correct? >

RE: Using dynamic allocation and shuffle service in Standalone Mode

2016-03-08 Thread Silvio Fiorito
You’ve started the external shuffle service on all worker nodes, correct? Can you confirm they’re still running and haven’t exited? From: Yuval.Itzchakov Sent: Tuesday, March 8, 2016 12:41 PM To: user@spark.apache.org Subject: Using