Memory tuning in spark sql

Hi Sir/madam, Greetings of the day. I am working on Spark 1.6.0 with AWS EMR(Elastic Map Reduce). I'm facing some issues in reading large(500 mb) file in spark-sql. Sometimes i get heap space error and sometimes the executors fail. i have increased the driver memory, executor memory, kryo

Re: Memory tuning in spark sql

ioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadE

Re: Memory tuning in spark sql

Can you provide a bit more detail ? values of the parameters you have tuned log snippets from executors snippet of your code Thanks On Thu, Feb 4, 2016 at 9:48 AM, wrote: > Hi Sir/madam, > Greetings of the day. > > I am working on Spark 1.6.0 with AWS EMR(Elastic