I have 5 workers each executor-memory 8GB of memory. My driver memory is 8 GB as well. They are all 8 core machines.
To answer Imran's question my configurations are thus. executor_total_max_heapsize = 18GB This problem happens at the end of my program. I don't have to run a lot of jobs to see this behaviour. I can see my output correctly in HDFS and all. I will give it one more try after increasing master's memory(which is default 296MB to 512 MB)
