Hi All,
I have been facing memory issues in spark. im using spark-sql on AWS EMR. i
have around 50GB file in AWS S3. I want to read this file in BI tool connected
to spark-sql on thrift-server over OBDC. I'm executing select * from table in
BI tool(qlikview,tableau).
I run into OOM error someti
Hi All,
Even i have same issues.
EMR conf is 3 node cluster with m3.2xlarge.
i'm tyring to read 100Gb file in spark-sql
i have set below on spark
export SPARK_EXECUTOR_MEMORY=4G
export SPARK_DRIVER_MEMORY=12G
export SPARK_EXECUTOR_INSTANCES=16
export SPARK_EXECUTOR_CORES=16
spark.kryoseriali
Hi Sir/madam,
Greetings of the day.
I am working on Spark 1.6.0 with AWS EMR(Elastic Map Reduce). I'm facing some
issues in reading large(500 mb) file in spark-sql.
Sometimes i get heap space error and sometimes the executors fail.
i have increased the driver memory, executor memory, kryo seriali
