I am trying to read in a file (4GB file). I tried setting both "spark.driver.memory" and "spark.executor.memory" to large values (say 16GB) but I still get a GC limit exceeded error. Any idea what I am missing?
On Mon, May 4, 2015 at 5:30 PM, Andrew Ash <[email protected]> wrote: > It's unlikely you need to increase the amount of memory on your master > node since it does simple bookkeeping. The majority of the memory pressure > across a cluster is on executor nodes. > > See the conf/spark-env.sh file for configuring heap sizes, and this > section in the docs for more information on how to make these changes: > http://spark.apache.org/docs/latest/configuration.html > > On Mon, May 4, 2015 at 2:24 PM, Vijayasarathy Kannan <[email protected]> > wrote: > >> Starting the master with "/sbin/start-master.sh" creates a JVM with only >> 512MB of memory. How to change this default amount of memory? >> >> Thanks, >> Vijay >> > >
