[
https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davies Liu reassigned SPARK-10342:
----------------------------------
Assignee: Davies Liu
> Cooperative memory management
> -----------------------------
>
> Key: SPARK-10342
> URL: https://issues.apache.org/jira/browse/SPARK-10342
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
> Affects Versions: 1.5.0
> Reporter: Davies Liu
> Assignee: Davies Liu
> Priority: Critical
>
> We have memory starving problems for a long time, it become worser in 1.5
> since we use larger page.
> In order to increase the memory usage (reduce unnecessary spilling) also
> reduce the risk of OOM, we should manage the memory in a cooperative way, it
> means all the memory consume should be also responsive to release memory
> (spilling) upon others' requests.
> The requests of memory could be different, hard requirement (will crash if
> not allocated) or soft requirement (worse performance if not allocated). Also
> the costs of spilling are also different. We could introduce some kind of
> priority to make them work together better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
