either increase overall executor memory if you have scope. or try to give more % to overhead memory from default of .7.
Read this <https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/> for more details. On Wed, Aug 2, 2017 at 11:03 PM Chetan Khatri <chetan.opensou...@gmail.com> wrote: > Can anyone please guide me with above issue. > > > On Wed, Aug 2, 2017 at 6:28 PM, Chetan Khatri <chetan.opensou...@gmail.com > > wrote: > >> Hello Spark Users, >> >> I have Hbase table reading and writing to Hive managed table where i >> applied partitioning by date column which worked fine but it has generate >> more number of files in almost 700 partitions but i wanted to use >> reparation to reduce File I/O by reducing number of files inside each >> partition. >> >> *But i ended up with below exception:* >> >> ExecutorLostFailure (executor 11 exited caused by one of the running >> tasks) Reason: Container killed by YARN for exceeding memory limits. 14.0 >> GB of 14 GB physical memory used. Consider boosting spark.yarn.executor. >> memoryOverhead. >> >> Driver memory=4g, executor mem=12g, num-executors=8, executor core=8 >> >> Do you think below setting can help me to overcome above issue: >> >> spark.default.parellism=1000 >> spark.sql.shuffle.partitions=1000 >> >> Because default max number of partitions are 1000. >> >> >> >
