Why would the spark master run out of RAM if I have too many slaves? Is this a flaw in the coding? I'm just a user of spark. The developer that set this up left the company, so I'm starting from the top here.
So I noticed if I spawn lots of jobs, my spark master ends up crashing due to low memory. It makes sense to me the master would just be a brain/controller to dish out jobs and the resources on the slaves is what would get used up, not the master. Thanks for any ideas/concepts/info. Appreciate it much
