It doesn't mean 'out of memory'; it means 'you can't allocate a byte[] over 2GB in the JVM'. Something is serializing a huge block somewhere. there are a number of related JIRAs and discussions on JIRA and this mailing list; have a browse of those first for back story.
On Mon, Feb 23, 2015 at 6:44 PM, insperatum <[email protected]> wrote: > Hi, I'm using MLLib to train a random forest. It's working fine to depth 15, > but if I use depth 20 I get a java.lang.OutOfMemoryError: Requested array > size exceeds VM limit on the driver, from the collectAsMap operation in > DecisionTree.scala, around line 642. It doesn't happen until a good hour > into training. I'm using 50 treees on 36 slaves with maxMemoryInMB=250, but > still get an error even if I use a driver memory of 240G. Has anybody seen > this error in this context before, and can advise on what might be > triggering it? Best, Luke > ________________________________ > View this message in context: Requested array size exceeds VM limit > Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
