Hi,
On Sat, Jan 17, 2015 at 3:05 AM, Shuai Zheng wrote:
>
> Can you share more information about how do you do that? I also have
> similar question about this.
>
Not very proud about it ;-), but here you go:
// find the number of workers available to us.
val _runCmd = scala.util.Properties.prop
: Re: Determine number of running executors
Hi,
Thanks for your help!
Sandy, I had a bit of trouble finding the spark.executor.cores property. (It
wasn't there although its value should have been 2.)
I ended up throwing regular expressions on
scala.util.Properties.propO
Hi,
Thanks for your help!
Sandy, I had a bit of trouble finding the spark.executor.cores property.
(It wasn't there although its value should have been 2.)
I ended up throwing regular expressions
on scala.util.Properties.propOrElse("sun.java.command", ""), which worked
surprisingly well ;-)
Than
Hi Tobias,
One way to find out the number of executors is through
SparkContext#getExecutorMemoryStatus. You can find out the number of by
asking the SparkConf for the "spark.executor.cores" property, which, if not
set, means 1 for YARN.
-Sandy
On Fri, Nov 21, 2014 at 1:30 AM, Yanbo Liang wrot
You can get parameter such as spark.executor.memory, but you can not get
executor or core numbers.
Because executor and core are parameters of spark deploy environment not
spark context.
val conf = new SparkConf().set("spark.executor.memory","2G")
val sc = new SparkContext(conf)
sc.getConf.get("s