It depends on how you use Spark, if you use Spark with Yarn and enable
dynamic allocation, the number of executor is not fixed, will change
dynamically according to the load.
Thanks
Jerry
2015-05-27 14:44 GMT+08:00 canan chen ccn...@gmail.com:
It seems the executor number is fixed for the
How does the dynamic allocation works ? I mean does it related
with parallelism of my RDD and how does driver know how many executor it
needs ?
On Wed, May 27, 2015 at 2:49 PM, Saisai Shao sai.sai.s...@gmail.com wrote:
It depends on how you use Spark, if you use Spark with Yarn and enable
The drive has a heuristic mechanism to decide the number of executors in
the run-time according the pending tasks. You could enable with
configuration, you could refer to spark document to find the details.
2015-05-27 15:00 GMT+08:00 canan chen ccn...@gmail.com:
How does the dynamic allocation
If with mesos, how do we control the number of executors? In our cluster,
each node only has one executor with very big JVM. Sometimes, if the
executor dies, all the concurrent running tasks will be gone. We would like
to have multiple executors in one node but can not figure out a way to do
it in
Typo. We can not figure a way to increase the number of executor in one
node in mesos.
On Wednesday, May 27, 2015, DB Tsai dbt...@dbtsai.com wrote:
If with mesos, how do we control the number of executors? In our cluster,
each node only has one executor with very big JVM. Sometimes, if the
I'm not sure about Mesos, maybe someone has the Mesos experience can help
answer this.
Thanks
Jerry
2015-05-27 15:21 GMT+08:00 DB Tsai dbt...@dbtsai.com:
Typo. We can not figure a way to increase the number of executor in one
node in mesos.
On Wednesday, May 27, 2015, DB Tsai
