Github user aarondav commented on the pull request:
https://github.com/apache/incubator-spark/pull/615#issuecomment-35818768
Updated to make de-publicize SPARK_DRIVER_MEMORY. `spark-class` is
currently only used in the following cases:
- spark-shell (which has a memory command-line option that under the hood
uses SPARK_DRIVER_MEMORY)
- developer examples (e.g., StoragePerfTester and FaultToleranceTest;
probably do not need memory configuration)
- launching Mesos executors (SPARK_EXECUTOR_MEMORY is used here)
- launching standalone workers and masters (SPARK_MASTER/WORKER_MEMORY is
used here)
- and starting YARN and Standalone "Clients" (memory probably does not need
to be set here, as the Client just sends a request to start a driver within the
cluster, and driver memory is set via command line options)
The last point is the most arguable, if for some reason YARN or Standalone
Clients may need more than 512 MB. We could always introduce a public
SPARK_CLIENT_MEMORY that is used only when launching YARN and Standalone
Clients, but I am not sure it's necessary. Otherwise, all of our cases seem
covered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastruct...@apache.org or file a JIRA ticket with INFRA.
---
