Hello Spark Users, I have Hbase table reading and writing to Hive managed table where i applied partitioning by date column which worked fine but it has generate more number of files in almost 700 partitions but i wanted to use reparation to reduce File I/O by reducing number of files inside each partition.
*But i ended up with below exception:* ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 14.0 GB of 14 GB physical memory used. Consider boosting spark.yarn.executor. memoryOverhead. Driver memory=4g, executor mem=12g, num-executors=8, executor core=8 Do you think below setting can help me to overcome above issue: spark.default.parellism=1000 spark.sql.shuffle.partitions=1000 Because default max number of partitions are 1000.
