liupengcheng created SPARK-26750:
------------------------------------
Summary: Estimate memory overhead with multi-cores
Key: SPARK-26750
URL: https://issues.apache.org/jira/browse/SPARK-26750
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 2.4.0
Reporter: liupengcheng
Currently, spark esitmate the memory overhead without taking multi-cores into
account, sometimes, it might cause direct memory oom, or killed by yarn for
exceeding requested physical memory.
I think the memory overhead is related to the executor's core number(mainly the
spark direct memory and some related jvm native memory, for instance, the
thread stacks, GC data etc.). so maybe we can improve this estimation by taking
the core number into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
