On Fri, May 15, 2015 at 5:09 PM, Thomas Gerber <thomas.ger...@radius.com> wrote:
> Now, we noticed that we get java heap OOM exceptions on the output tracker > when we have too many tasks. I wonder: > 1. where does the map output tracker live? The driver? The master (when > those are not the same)? > 2. how can we increase the heap for it? Especially when using spark-submit? > It does not live on the master -- that is only in a standalone cluster, and it does very little work. (Though there are *Master and *Worker variants of the class, its really running on the driver and the executors.) If you are getting OOMs in the MapOutputTrackerMaster (which lives on the driver), then you can increase the memory for the driver via the normal args for controlling driver memory, with "--driver-memory 10G" or whatever. Just to be clear, if you hit an OOM from somewhere in the MapOutputTracker code, it just means that code is what pushed things over the top. Of course you could have 99% of your memory used by something else, perhaps your own data structures, which perhaps could be trimmed down. You could get a heap dump on the driver to see where the memory is really getting used. Do you mind sharing the details of how you hit these OOMs? How much memory, how many partitions on each side of the shuffle? Sort based shuffle I assume? thanks, Imran
