Hello, I am trying to execute a query that joins two large tables (3 million and 20 million records). I am getting the Hadoop error code 2 during execution. This happens mainly while the reducers are running. Sometimes the reducers complete 100% and then the error comes. The logs talk about running out of Heap space and GC overhead limit exceeding.
I am running a 6 node cluster with child JVM memory of 1GB. Are there any parameters I could tweak to make them run? Is adding more nodes the solution to such problem? Thanks!
