Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/21589
> Unless there is some other compelling reason for introducing this which I
have missed; I am -1 on introducing this change.
I would like to describe one class of use cases which you don't consider
seriously for some reasons. You are mostly talking about the cases when a
cluster is shared among many apps/users/jobs, and not all resources are
available to submitted jobs. In that cases, the proposed methods are useless no
doubt. But there is another trend nowadays.
Creating a cluster is becoming pretty cheap. The process takes a few
seconds. Our clients create new cluster per one job, and typical use cases when
one jobs occupies all cluster resources. A cluster become like a container for
one job. Analogy between Virtual Machines and containers here is direct. I
would like you look at this use cases more seriously. Users can spin up new
cluster for any activity in their app - one for machine/deep learning
(`numExecutors` is useful here), another one for crunching inputs (fine tuning
of CPU usage is need here). I believe our users/customers are smart enough to
use the proposed methods in right way.
Can we add the methods as experimental and if we will observe some problems
in the upcoming releases, we will just remove them? /cc @gatorsmile @rxin
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
