if you want you can determine the number of executor as well
by setting 'spark.executor.instances' property in 'sparkConf' object.
Thank you.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23241.html
Sent from
Note that this property is only available for YARN
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23256.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
from Samsung Mobile
Original message
From: maxdml
Date:2015/06/10 19:56 (GMT+00:00)
To: user@spark.apache.org
Subject: Re: Determining number of executors within RDD
Actually this is somehow confusing for two reasons:
- First, the option 'spark.executor.instances', which
/executot
Sent from Samsung Mobile
div Original message /divdivFrom: maxdml
max...@cs.duke.edu /divdivDate:2015/06/10 19:56 (GMT+00:00)
/divdivTo: user@spark.apache.org /divdivSubject: Re: Determining number
of executors within RDD /divdiv
/divActually this is somehow confusing
-td13692.html)
Could anyone clarify this? :-)
Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23262.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
workers per machine
Sent from Samsung Mobile
Original message
From: Sandy Ryza
Date:2015/06/10 21:31 (GMT+00:00)
To: Evo Eftimov
Cc: maxdml ,user@spark.apache.org
Subject: Re: Determining number of executors within RDD
On YARN, there is no concept of a Spark Worker
.n3.nabble.com/Determining-number-of-executors-within-RDD-tp15554p23234.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional
Hi,
I want implement an RDD wherein the decision of number of partitions is
based on the number of executors that have been set up. Is there some way I
can determine the number of executors within the getPartitions() call?
