I filed https://issues.apache.org/jira/browse/SPARK-3884 to address this.
-Sandy On Thu, Oct 9, 2014 at 7:05 AM, Greg Hill <greg.h...@rackspace.com> wrote: > $MASTER is 'yarn-cluster' in spark-env.sh > > spark-submit --driver-memory 12424m --class > org.apache.spark.examples.SparkPi > /usr/lib/spark-yarn/lib/spark-examples*.jar 1000 > OpenJDK 64-Bit Server VM warning: INFO: > os::commit_memory(0x00000006fd280000, 4342677504, 0) failed; > error='Cannot allocate memory' (errno=12) > # > # There is insufficient memory for the Java Runtime Environment to > continue. > # Native memory allocation (malloc) failed to allocate 4342677504 bytes > for committing reserved memory. > # An error report file with more information is saved as: > # /tmp/jvm-3525/hs_error.log > > > From: Andrew Or <and...@databricks.com> > Date: Wednesday, October 8, 2014 3:25 PM > To: Greg <greg.h...@rackspace.com> > Cc: "user@spark.apache.org" <user@spark.apache.org> > Subject: Re: Spark on YARN driver memory allocation bug? > > Hi Greg, > > It does seem like a bug. What is the particular exception message that > you see? > > Andrew > > 2014-10-08 12:12 GMT-07:00 Greg Hill <greg.h...@rackspace.com>: > >> So, I think this is a bug, but I wanted to get some feedback before I >> reported it as such. On Spark on YARN, 1.1.0, if you specify the >> --driver-memory value to be higher than the memory available on the client >> machine, Spark errors out due to failing to allocate enough memory. This >> happens even in yarn-cluster mode. Shouldn't it only allocate that memory >> on the YARN node that is going to run the driver process, not the local >> client machine? >> >> Greg >> >> >
