GitHub user aarondav opened a pull request:
https://github.com/apache/spark/pull/99
SPARK-929: Fully deprecate usage of SPARK_MEM
(Continued from old repo, prior discussion at
https://github.com/apache/incubator-spark/pull/615)
This patch cements our deprecation of the SPARK_MEM environment variable by
replacing it with three more specialized variables:
SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_DRIVER_MEMORY
The creation of the latter two variables means that we can safely set
driver/job memory without accidentally setting the executor memory. Neither is
public.
SPARK_EXECUTOR_MEMORY is only used by the Mesos scheduler (and set within
SparkContext). The proper way of configuring executor memory is through the
"spark.executor.memory" property.
SPARK_DRIVER_MEMORY is the new way of specifying the amount of memory run
by jobs launched by spark-class, without possibly affecting executor memory.
Other memory considerations:
- The repl's memory can be set through the "--drivermem" command-line
option, which really just sets SPARK_DRIVER_MEMORY.
- run-example doesn't use spark-class, so the only way to modify examples'
memory is actually an unusual use of SPARK_JAVA_OPTS (which is normally
overriden in all cases by spark-class).
This patch also fixes a lurking bug where spark-shell misused spark-class
(the first argument is supposed to be the main class name, not java options),
as well as a bug in the Windows spark-class2.cmd. I have not yet tested this
patch on either Windows or Mesos, however.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aarondav/spark sparkmem
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/99.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #99
----
commit 9df4c68262dac0edde1ec5bdd1fd065d2bf34e00
Author: Aaron Davidson <[email protected]>
Date: 2014-02-17T23:09:51Z
SPARK-929: Fully deprecate usage of SPARK_MEM
This patch cements our deprecation of the SPARK_MEM environment variable
by replacing it with case-specific variables:
SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_DRIVER_MEMORY
The creation of the latter two variables means that we can safely
set driver/job memory without accidentally setting the executor memory.
Neither is public.
SPARK_EXECUTOR_MEMORY is only used by the Mesos scheduler (and set
within SparkContext). The proper way of configuring executor memory
is through the "spark.executor.memory" property.
SPARK_DRIVER_MEMORY is the new way of specifying the amount of memory
run by jobs launched by spark-class, without possibly affecting executor
memory.
Other memory considerations:
- The repl's memory can be set through the "--drivermem" command-line
option,
which really just sets SPARK_DRIVER_MEMORY.
- run-example doesn't use spark-class, so the only way to modify examples'
memory is actually an unusual use of SPARK_JAVA_OPTS (which is normally
overriden in all cases by spark-class).
This patch also fixes a lurking bug where spark-shell misused spark-class
(the first argument is supposed to be the main class name, not java
options).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
