Hi,
Yes, this is a problem, and I'm not aware of any simple workarounds
(or complex one for that matter). There are people working to fix
this, you can follow progress here:
https://issues.apache.org/jira/browse/SPARK-1239
On Tue, Sep 9, 2014 at 2:54 PM, jbeynon jbey...@gmail.com wrote:
I'm running on Yarn with relatively small instances with 4gb memory. I'm not
caching any data but when the map stage ends and shuffling begins all of the
executors request the map output locations at the same time which seems to
kill the driver when the number of executors is turned up.
For example, the size of output statuses is about 10mb and with 500
executors the driver appears to be making 500 (5gb of data) copies of this
data to send out and running out of memory. When I turn down the number of
executors everything runs fine.
Has anyone else run into this? Maybe I'm misunderstanding the underlying
cause. I don't have a copy of the stack trace handy but can recreate it if
necessary. It was somewhere in the init for HeapByteBuffer. Any advice
would be helpful.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Driver-OOME-Java-heap-space-when-executors-request-map-output-locations-tp13827.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
--
Marcelo
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
