[
https://issues.apache.org/jira/browse/SPARK-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tobias Bertelsen updated SPARK-5349:
------------------------------------
Description:
The resource requirements of an interactive shell varies heavily. Sometimes
heavy commands are executed, and sometimes the user is thinking, getting
coffee, interrupted etc...
A spark shell allocates a fixed number of worker cores (at least in standalone
mode). A user thus has the choice to either block other users from the cluster
by allocating all cores (default behavior), or restrict him/herself to only a
few cores using the option {{--total-executor-cores}}. Either way the cores
allocated to the shell has low utilization, since they will be waiting for the
user a lot.
Instead the spark shell allocate resources directly required to run the driver,
and request worker cores only when computation is performed on the RDDs.
This should allow for multiple users, to use an interactive shell concurrently
while stille utilizing the entire cluster, when performing heavy operations.
was:
The documentation states
> Multiple spark shells should be able to share resources
> -------------------------------------------------------
>
> Key: SPARK-5349
> URL: https://issues.apache.org/jira/browse/SPARK-5349
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Tobias Bertelsen
>
> The resource requirements of an interactive shell varies heavily. Sometimes
> heavy commands are executed, and sometimes the user is thinking, getting
> coffee, interrupted etc...
> A spark shell allocates a fixed number of worker cores (at least in
> standalone mode). A user thus has the choice to either block other users from
> the cluster by allocating all cores (default behavior), or restrict
> him/herself to only a few cores using the option {{--total-executor-cores}}.
> Either way the cores allocated to the shell has low utilization, since they
> will be waiting for the user a lot.
> Instead the spark shell allocate resources directly required to run the
> driver, and request worker cores only when computation is performed on the
> RDDs.
> This should allow for multiple users, to use an interactive shell
> concurrently while stille utilizing the entire cluster, when performing heavy
> operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
