Give your config the cluster manager can only give 2 Executors. Looking at m3.2xlarge --> its is with 30 GB Memory . you have 3 *m3.2xlarge which means you have total of 3 * 30 Gb memory for executor. 15 GB for 16 executor would require 15 * 16 GB. Also check executor the number of core you are setting.
Kuchekar, Nilesh On Wed, Feb 17, 2016 at 8:02 PM, <arun.bong...@cognizant.com> wrote: > Hi All, > > I have been facing memory issues in spark. im using spark-sql on AWS EMR. > i have around 50GB file in AWS S3. I want to read this file in BI tool > connected to spark-sql on thrift-server over OBDC. I'm executing select * > from table in BI tool(qlikview,tableau). > I run into OOM error sometimes and some time the LOST_EXECUTOR. I'm really > confused. > The spark runs fine for smaller data set. > > I have 3 node EMR cluster with m3.2xlarge. > > I have set below conf on spark. > > export SPARK_EXECUTOR_INSTANCES=16 > export SPARK_EXECUTOR_CORES=16 > export SPARK_EXECUTOR_MEMORY=15G > export SPARK_DRIVER_MEMORY=12G > spark.kryoserializer.buffer.max 1024m > > Even after setting SPARK_EXECUTOR_INSTANCES as 16, only 2 executors come > up. > > This is been road block since long time. Any help would be appreciated. > > Thanks > Arun. > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. If you are not the intended recipient(s), please reply to the > sender and destroy all copies of the original message. Any unauthorized > review, use, disclosure, dissemination, forwarding, printing or copying of > this email, and/or any action taken in reliance on the contents of this > e-mail is strictly prohibited and may be unlawful. Where permitted by > applicable law, this e-mail and other e-mail communications sent to and > from Cognizant e-mail addresses may be monitored. >
