ok yes, I think I have narrowed it down to being a problem with driver memory settings. It looks like the application master/driver is not being launched with the settings specified:
For the driver process on the main node I see "-XX:MaxPermSize=128m -Xms512m -Xmx512m" as options used to start the JVM, even though I specified 'spark.yarn.am.memory', '5g' 'spark.yarn.am.memoryOverhead', '2000' The info shows that these options were read: 15/04/23 13:47:47 INFO yarn.Client: Will allocate AM container, with 7120 MB memory including 2000 MB overhead Is there some reason why these options are being ignored and instead starting the driver with just 512Mb of heap? On Thu, Apr 23, 2015 at 8:06 AM, Rok Roskar <rokros...@gmail.com> wrote: > the feature dimension is 800k. > > yes, I believe the driver memory is likely the problem since it doesn't > crash until the very last part of the tree aggregation. > > I'm running it via pyspark through YARN -- I have to run in client mode so > I can't set spark.driver.memory -- I've tried setting the > spark.yarn.am.memory and overhead parameters but it doesn't seem to have an > effect. > > Thanks, > > Rok > > On Apr 23, 2015, at 7:47 AM, Xiangrui Meng <men...@gmail.com> wrote: > > > What is the feature dimension? Did you set the driver memory? -Xiangrui > > > > On Tue, Apr 21, 2015 at 6:59 AM, rok <rokros...@gmail.com> wrote: > >> I'm trying to use the StandardScaler in pyspark on a relatively small > (a few > >> hundred Mb) dataset of sparse vectors with 800k features. The fit > method of > >> StandardScaler crashes with Java heap space or Direct buffer memory > errors. > >> There should be plenty of memory around -- 10 executors with 2 cores > each > >> and 8 Gb per core. I'm giving the executors 9g of memory and have also > tried > >> lots of overhead (3g), thinking it might be the array creation in the > >> aggregators that's causing issues. > >> > >> The bizarre thing is that this isn't always reproducible -- sometimes it > >> actually works without problems. Should I be setting up executors > >> differently? > >> > >> Thanks, > >> > >> Rok > >> > >> > >> > >> > >> -- > >> View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/StandardScaler-failing-with-OOM-errors-in-PySpark-tp22593.html > >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > >> For additional commands, e-mail: user-h...@spark.apache.org > >> > >
