Can anyone please guide me with above issue.
On Wed, Aug 2, 2017 at 6:28 PM, Chetan Khatri <chetan.opensou...@gmail.com> wrote: > Hello Spark Users, > > I have Hbase table reading and writing to Hive managed table where i > applied partitioning by date column which worked fine but it has generate > more number of files in almost 700 partitions but i wanted to use > reparation to reduce File I/O by reducing number of files inside each > partition. > > *But i ended up with below exception:* > > ExecutorLostFailure (executor 11 exited caused by one of the running > tasks) Reason: Container killed by YARN for exceeding memory limits. 14.0 > GB of 14 GB physical memory used. Consider boosting spark.yarn.executor. > memoryOverhead. > > Driver memory=4g, executor mem=12g, num-executors=8, executor core=8 > > Do you think below setting can help me to overcome above issue: > > spark.default.parellism=1000 > spark.sql.shuffle.partitions=1000 > > Because default max number of partitions are 1000. > > >
