GitHub user ejono opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/774
[ZEPPELIN-735] Remove spark.executor.memory default to 512m
### What is this PR for?
The Spark interpreter currently honors whatever is set for
spark.executor.memory in spark-defaults.conf upon startup, but if you look at
the Interpreter page, you'll see that it has a default of 512m. If you restart
a running Spark interpreter from this page, the new SparkContext will use this
new default of spark.executor.memory=512m instead of what it had previously
pulled from spark-defaults.conf.
Removing this 512m default from SparkInterpreter code will allow
spark.executor.memory to default to whatever value may be set in
spark-defaults.conf, falling back to the Spark built-in default (which, btw,
has for a few Spark versions been 1g, not 512m anymore).
### What type of PR is it?
Improvement
### Todos
N/A
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-735
### How should this be tested?
* Set spark.executor.memory to some value in spark-defaults.conf (say,
5120m)
* Run a Spark paragraph in Zeppelin
** The Spark application will correctly use the spark.executor.memory value
from spark-defaults.conf (both before and after this change).
* View Interpreter page in Zeppelin UI
** Before this change: spark.executor.memory will be displayed as 512m
instead of what is in spark-defaults.conf
** After this change: spark.executor.memory will be blank
* Restart Spark Interpreter from this page and run a Spark paragraph
** Before this change: the new Spark application will incorrectly use the
spark.executor.memory=512m value shown on the Interpreter page
** After this change: the new Spark application will correctly use the
spark.executor.memory value from spark-defaults.conf
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ejono/incubator-zeppelin ZEPPELIN-735
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/774.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #774
----
commit 3c68e095cc8245267ab72b56f5f366732fd35b20
Author: Jonathan Kelly <jonat...@amazon.com>
Date: 2016-03-11T19:29:39Z
Remove spark.executor.memory default to 512m
This enables spark.executor.memory to default to whatever value may be set
in
spark-defaults.conf, falling back to the Spark built-in default (which has
for a
few versions been 1g, not 512m).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
