Hi Sir/madam, Greetings of the day. I am working on Spark 1.6.0 with AWS EMR(Elastic Map Reduce). I'm facing some issues in reading large(500 mb) file in spark-sql. Sometimes i get heap space error and sometimes the executors fail. i have increased the driver memory, executor memory, kryo serializer buffer size.. etc.. but nothing helps.
I Kindly request your help in resolving this issue. Thanks Arun. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
