Can you add more details like are you using rdds/datasets/sql ..; are you
doing group by/ joins ; is your input splittable?
btw, you can pass the config the same way you are passing memryOverhead:
e.g.
--conf spark.default.parallelism=1000 or through spark-context in code
Regards,
Sushrut Ikhar
Hi All,
Any updates on this?
On Wednesday 28 September 2016 12:22 PM, Sushrut Ikhar wrote:
Try with increasing the parallelism by repartitioning and also you may
increase - spark.default.parallelism
You can also try with decreasing num-executor cores.
Basically, this happens when the executor
:
Thanks Sushrut for the reply.
Currently I have not defined spark.default.parallelism property.
Can you let me know how much should I set it to?
Regards,
Aditya Calangutkar
On Wednesday 28 September 2016 12:22 PM, Sushrut Ikhar wrote:
Try with increasing the parallelism by repartitioning
I have a spark job which runs fine for small data. But when data
increases it gives executor lost error.My executor and driver memory are
set at its highest point. I have also tried increasing--conf
spark.yarn.executor.memoryOverhead=600but still not able to fix the
problem. Is there any other
