Davies Liu created SPARK-10342:
----------------------------------
Summary: Cooperative memory management
Key: SPARK-10342
URL: https://issues.apache.org/jira/browse/SPARK-10342
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 1.5.0
Reporter: Davies Liu
Assignee: Davies Liu
Priority: Critical
We have memory starving problems for a long time, it become worser in 1.5 since
we use larger page.
In order to increase the memory usage (reduce unnecessary spilling) also reduce
the risk of OOM, we should manage the memory in a cooperative way, it means all
the memory consume should be also responsive to release memory (spilling) upon
others' requests.
The requests of memory could be different, hard requirement (will crash if not
allocated) or soft requirement (worse performance if not allocated). Also the
costs of spilling are also different. We could introduce some kind of priority
to make them work together better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
