One possible cause of a standalone master OOMing is https://issues.apache.org/jira/browse/SPARK-6270. In 2.x, this will be fixed by https://issues.apache.org/jira/browse/SPARK-12299. In 1.x, one mitigation is to disable event logging. Another workaround would be to produce a patch which disables eager UI reconstruction for 1.x.
On Thu, Mar 31, 2016 at 11:16 AM Dillian Murphey <crackshotm...@gmail.com> wrote: > Why would the spark master run out of RAM if I have too many slaves? Is > this a flaw in the coding? I'm just a user of spark. The developer that > set this up left the company, so I'm starting from the top here. > > So I noticed if I spawn lots of jobs, my spark master ends up crashing due > to low memory. It makes sense to me the master would just be a > brain/controller to dish out jobs and the resources on the slaves is what > would get used up, not the master. > > Thanks for any ideas/concepts/info. > > Appreciate it much > >
