Re: Spark 1.5.1 Dynamic Resource Allocation
Hi Tom,
I believe a workaround is to set `spark.dynamicAllocation.initialExecutors`
to 0. As others have mentioned, from Spark 1.5.2 onwards this should no
longer be necessary.
-Andrew
2015-11-09 8:19 GMT-08:00 Jonathan Kelly :
> Tom,
>
> You might be hitting https://issues.apache.org/jira/browse/SPARK-10790,
> which was introduced in Spark 1.5.0 and fixed in 1.5.2. Spark 1.5.2 just
> passed release candidate voting, so it should be tagged, released and
> announced soon. If you are able to build from source yourself and run with
> that, you might want to try building from the v1.5.2-rc2 tag to see if it
> fixes your issue. Otherwise, hopefully Spark 1.5.2 will be available for
> download very soon.
>
> ~ Jonathan
>
> On Mon, Nov 9, 2015 at 6:08 AM, Akhil Das
> wrote:
>
>> Did you go through
>> http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup
>> for yarn, i guess you will have to copy the spark-1.5.1-yarn-shuffle.jar to
>> the classpath of all nodemanagers in your cluster.
>>
>> Thanks
>> Best Regards
>>
>> On Fri, Oct 30, 2015 at 7:41 PM, Tom Stewart <
>> stewartthom...@yahoo.com.invalid> wrote:
>>
>>> I am running the following command on a Hadoop cluster to launch Spark
>>> shell with DRA:
>>> spark-shell --conf spark.dynamicAllocation.enabled=true --conf
>>> spark.shuffle.service.enabled=true --conf
>>> spark.dynamicAllocation.minExecutors=4 --conf
>>> spark.dynamicAllocation.maxExecutors=12 --conf
>>> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=120 --conf
>>> spark.dynamicAllocation.schedulerBacklogTimeout=300 --conf
>>> spark.dynamicAllocation.executorIdleTimeout=60 --executor-memory 512m
>>> --master yarn-client --queue default
>>>
>>> This is the code I'm running within the Spark Shell - just demo stuff
>>> from teh web site.
>>>
>>> import org.apache.spark.mllib.clustering.KMeans
>>> import org.apache.spark.mllib.linalg.Vectors
>>>
>>> // Load and parse the data
>>> val data = sc.textFile("hdfs://ns/public/sample/kmeans_data.txt")
>>>
>>> val parsedData = data.map(s => Vectors.dense(s.split('
>>> ').map(_.toDouble))).cache()
>>>
>>> // Cluster the data into two classes using KMeans
>>> val numClusters = 2
>>> val numIterations = 20
>>> val clusters = KMeans.train(parsedData, numClusters, numIterations)
>>>
>>> This works fine on Spark 1.4.1 but is failing on Spark 1.5.1. Did
>>> something change that I need to do differently for DRA on 1.5.1?
>>>
>>> This is the error I am getting:
>>> 15/10/29 21:44:19 WARN YarnScheduler: Initial job has not accepted any
>>> resources; check your cluster UI to ensure that workers are registered and
>>> have sufficient resources
>>> 15/10/29 21:44:34 WARN YarnScheduler: Initial job has not accepted any
>>> resources; check your cluster UI to ensure that workers are registered and
>>> have sufficient resources
>>> 15/10/29 21:44:49 WARN YarnScheduler: Initial job has not accepted any
>>> resources; check your cluster UI to ensure that workers are registered and
>>> have sufficient resources
>>>
>>> That happens to be the same error you get if you haven't followed the
>>> steps to enable DRA, however I have done those and as I said if I just flip
>>> to Spark 1.4.1 on the same cluster it works with my YARN config.
>>>
>>>
>>
>
Re: Spark 1.5.1 Dynamic Resource Allocation
Tom,
You might be hitting https://issues.apache.org/jira/browse/SPARK-10790,
which was introduced in Spark 1.5.0 and fixed in 1.5.2. Spark 1.5.2 just
passed release candidate voting, so it should be tagged, released and
announced soon. If you are able to build from source yourself and run with
that, you might want to try building from the v1.5.2-rc2 tag to see if it
fixes your issue. Otherwise, hopefully Spark 1.5.2 will be available for
download very soon.
~ Jonathan
On Mon, Nov 9, 2015 at 6:08 AM, Akhil Das
wrote:
> Did you go through
> http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup
> for yarn, i guess you will have to copy the spark-1.5.1-yarn-shuffle.jar to
> the classpath of all nodemanagers in your cluster.
>
> Thanks
> Best Regards
>
> On Fri, Oct 30, 2015 at 7:41 PM, Tom Stewart <
> stewartthom...@yahoo.com.invalid> wrote:
>
>> I am running the following command on a Hadoop cluster to launch Spark
>> shell with DRA:
>> spark-shell --conf spark.dynamicAllocation.enabled=true --conf
>> spark.shuffle.service.enabled=true --conf
>> spark.dynamicAllocation.minExecutors=4 --conf
>> spark.dynamicAllocation.maxExecutors=12 --conf
>> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=120 --conf
>> spark.dynamicAllocation.schedulerBacklogTimeout=300 --conf
>> spark.dynamicAllocation.executorIdleTimeout=60 --executor-memory 512m
>> --master yarn-client --queue default
>>
>> This is the code I'm running within the Spark Shell - just demo stuff
>> from teh web site.
>>
>> import org.apache.spark.mllib.clustering.KMeans
>> import org.apache.spark.mllib.linalg.Vectors
>>
>> // Load and parse the data
>> val data = sc.textFile("hdfs://ns/public/sample/kmeans_data.txt")
>>
>> val parsedData = data.map(s => Vectors.dense(s.split('
>> ').map(_.toDouble))).cache()
>>
>> // Cluster the data into two classes using KMeans
>> val numClusters = 2
>> val numIterations = 20
>> val clusters = KMeans.train(parsedData, numClusters, numIterations)
>>
>> This works fine on Spark 1.4.1 but is failing on Spark 1.5.1. Did
>> something change that I need to do differently for DRA on 1.5.1?
>>
>> This is the error I am getting:
>> 15/10/29 21:44:19 WARN YarnScheduler: Initial job has not accepted any
>> resources; check your cluster UI to ensure that workers are registered and
>> have sufficient resources
>> 15/10/29 21:44:34 WARN YarnScheduler: Initial job has not accepted any
>> resources; check your cluster UI to ensure that workers are registered and
>> have sufficient resources
>> 15/10/29 21:44:49 WARN YarnScheduler: Initial job has not accepted any
>> resources; check your cluster UI to ensure that workers are registered and
>> have sufficient resources
>>
>> That happens to be the same error you get if you haven't followed the
>> steps to enable DRA, however I have done those and as I said if I just flip
>> to Spark 1.4.1 on the same cluster it works with my YARN config.
>>
>>
>
Re: Spark 1.5.1 Dynamic Resource Allocation
Did you go through
http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup
for yarn, i guess you will have to copy the spark-1.5.1-yarn-shuffle.jar to
the classpath of all nodemanagers in your cluster.
Thanks
Best Regards
On Fri, Oct 30, 2015 at 7:41 PM, Tom Stewart <
stewartthom...@yahoo.com.invalid> wrote:
> I am running the following command on a Hadoop cluster to launch Spark
> shell with DRA:
> spark-shell --conf spark.dynamicAllocation.enabled=true --conf
> spark.shuffle.service.enabled=true --conf
> spark.dynamicAllocation.minExecutors=4 --conf
> spark.dynamicAllocation.maxExecutors=12 --conf
> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=120 --conf
> spark.dynamicAllocation.schedulerBacklogTimeout=300 --conf
> spark.dynamicAllocation.executorIdleTimeout=60 --executor-memory 512m
> --master yarn-client --queue default
>
> This is the code I'm running within the Spark Shell - just demo stuff from
> teh web site.
>
> import org.apache.spark.mllib.clustering.KMeans
> import org.apache.spark.mllib.linalg.Vectors
>
> // Load and parse the data
> val data = sc.textFile("hdfs://ns/public/sample/kmeans_data.txt")
>
> val parsedData = data.map(s => Vectors.dense(s.split('
> ').map(_.toDouble))).cache()
>
> // Cluster the data into two classes using KMeans
> val numClusters = 2
> val numIterations = 20
> val clusters = KMeans.train(parsedData, numClusters, numIterations)
>
> This works fine on Spark 1.4.1 but is failing on Spark 1.5.1. Did
> something change that I need to do differently for DRA on 1.5.1?
>
> This is the error I am getting:
> 15/10/29 21:44:19 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
> 15/10/29 21:44:34 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
> 15/10/29 21:44:49 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
>
> That happens to be the same error you get if you haven't followed the
> steps to enable DRA, however I have done those and as I said if I just flip
> to Spark 1.4.1 on the same cluster it works with my YARN config.
>
>
Re: Spark 1.5.1 Dynamic Resource Allocation
https://issues.apache.org/jira/browse/SPARK-10790 Changed to add minExecutors < initialExecutors < maxExecutors and that works. spark-shell --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.minExecutors=2 --conf spark.dynamicAllocation.initialExecutors=4 --conf spark.dynamicAllocation.maxExecutors=12 --conf spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=120 --conf spark.dynamicAllocation.schedulerBacklogTimeout=300 --conf spark.dynamicAllocation.executorIdleTimeout=60 --executor-memory 512m --master yarn-client --queue default -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-5-1-Dynamic-Resource-Allocation-tp25275p25277.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
