[ https://issues.apache.org/jira/browse/SPARK-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Davidson updated SPARK-1264: ---------------------------------- Assignee: (was: Aaron Davidson) > Documentation for setting heap sizes across all configurations > -------------------------------------------------------------- > > Key: SPARK-1264 > URL: https://issues.apache.org/jira/browse/SPARK-1264 > Project: Spark > Issue Type: Improvement > Components: Documentation > Affects Versions: 1.0.0 > Reporter: Andrew Ash > > As a user, there are lots of places to configure heap sizes, and it takes a > bit of trial and error to figure out how to configure what you want. > We need some more clear documentation on how set these for the cross product > of Spark components (master, worker, executor, driver, shell) and deployment > modes (Standalone, YARN, Mesos, EC2?). > I'm happy to do the authoring if someone can help pull together the relevant > details. > Here's the best I've got so far: > {noformat} > # Standalone cluster > Master - SPARK_DAEMON_MEMORY - default: 512mb > Worker - SPARK_DAEMON_MEMORY vs SPARK_WORKER_MEMORY? - default: ? See > WorkerArguments.inferDefaultMemory() > Executor - spark.executor.memory > Driver - SPARK_DRIVER_MEMORY - default: 512mb > Shell - A pre-built driver so SPARK_DRIVER_MEMORY - default: 512mb > # EC2 cluster > Master - ? > Worker - ? > Executor - ? > Driver - ? > Shell - ? > # Mesos cluster > Master - SPARK_DAEMON_MEMORY > Worker - SPARK_DAEMON_MEMORY > Executor - SPARK_EXECUTOR_MEMORY > Driver - SPARK_DRIVER_MEMORY > Shell - A pre-built driver so SPARK_DRIVER_MEMORY > # YARN cluster > Master - SPARK_MASTER_MEMORY ? > Worker - SPARK_WORKER_MEMORY ? > Executor - SPARK_EXECUTOR_MEMORY > Driver - SPARK_DRIVER_MEMORY > Shell - A pre-built driver so SPARK_DRIVER_MEMORY > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)