You don't in general configure Spark with environment variables. They exist but largely for backwards compatibility. Use arguments like --executor-memory on spark-submit, which are explained in the docs and the help message. It is possible to directly set the system properties with -D too if you need to do so directly. You don't have to change your app. Executor memory does not have to be set this way but you could. On Jan 1, 2015 6:36 AM, "Kevin Burton" <bur...@spinn3r.com> wrote:
> This is really weird and I’m surprised no one has found this issue yet. > > I’ve spent about an hour or more trying to debug this :-( > > My spark install is ignoring ALL my memory settings. And of course my job > is running out of memory. > > The default is 512MB so pretty darn small. > > The worker and master start up and both use 512M > > This alone is very weird and poor documentation IMO because: > > "SPARK_WORKER_MEMORY, to set how much total memory workers have to give > executors (e.g. 1000m, 2g)” > > … so if it’s giving it to executors, AKA the memory executors run with, > then it should be SPARK_EXECUTOR_MEMORY… > > … and the worker actually uses SPARK_DAEMON memory. > > but actually I’m right. It IS SPARK_EXECUTOR_MEMORY… according to > bin/spark-class > > … but, that’s not actually being used :-( > > that setting is just flat out begin ignored and it’s just using 512MB. So > all my jobs fail. > > … and I write an ‘echo’ so I could trace the spark-class script to see > what the daemons are actually being run with and spark-class wasn’t being > called with and nothing is logged for the coarse grained executor. I guess > it’s just inheriting the JVM opts from it’s parent and Java is launching > the process directly? > > This is a nightmare :( > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> > >
