Re: Spark job throwing “java.lang.OutOfMemoryError: GC overhead limit exceeded”

Hi Raj, Since the number of executor cores is equivalent to the number of tasks that can be executed in parallel in the executor, in effect, the 6G executor memory configured for an executor is being shared by 6 tasks plus factoring in the memory allocation for caching task execution. I would

Spark job throwing “java.lang.OutOfMemoryError: GC overhead limit exceeded”

Hello All, I have a Spark job that throws java.lang.OutOfMemoryError: GC overhead limit exceeded. The job is trying to process a filesize 4.5G. I've tried following spark configuration: --num-executors 6 --executor-memory 6G --executor-cores 6 --driver-memory 3G I tried increasing more