wow. Just figured it out:
conf.set( "spark.executor.memory", "2g");
I have to set it in the Job… that’s really counter intuitive. Especially
because the documentation in spark-env.sh says the exact opposite.
What’s the resolution here. This seems like a mess. I’d propose a solution
to clean it up but I don’t know where to begin.
On Wed, Dec 31, 2014 at 10:35 PM, Kevin Burton <bur...@spinn3r.com> wrote:
> This is really weird and I’m surprised no one has found this issue yet.
>
> I’ve spent about an hour or more trying to debug this :-(
>
> My spark install is ignoring ALL my memory settings. And of course my job
> is running out of memory.
>
> The default is 512MB so pretty darn small.
>
> The worker and master start up and both use 512M
>
> This alone is very weird and poor documentation IMO because:
>
> "SPARK_WORKER_MEMORY, to set how much total memory workers have to give
> executors (e.g. 1000m, 2g)”
>
> … so if it’s giving it to executors, AKA the memory executors run with,
> then it should be SPARK_EXECUTOR_MEMORY…
>
> … and the worker actually uses SPARK_DAEMON memory.
>
> but actually I’m right. It IS SPARK_EXECUTOR_MEMORY… according to
> bin/spark-class
>
> … but, that’s not actually being used :-(
>
> that setting is just flat out begin ignored and it’s just using 512MB. So
> all my jobs fail.
>
> … and I write an ‘echo’ so I could trace the spark-class script to see
> what the daemons are actually being run with and spark-class wasn’t being
> called with and nothing is logged for the coarse grained executor. I guess
> it’s just inheriting the JVM opts from it’s parent and Java is launching
> the process directly?
>
> This is a nightmare :(
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
