[
https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14947687#comment-14947687
]
Reynold Xin commented on SPARK-10342:
-------------------------------------
[~fxing] thanks a lot for the interest. Since this is your first time
contributing to Spark, it'd be better to start with some simpler tasks. This
task itself is fairly complicated and would require deep understanding of the
internals to do. I'd recommend searching for some starter tasks and bug fixes
to get yourself warmed up first, and then move towards the more challenging
ones.
> Cooperative memory management
> -----------------------------
>
> Key: SPARK-10342
> URL: https://issues.apache.org/jira/browse/SPARK-10342
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
> Affects Versions: 1.5.0
> Reporter: Davies Liu
> Priority: Critical
>
> We have memory starving problems for a long time, it become worser in 1.5
> since we use larger page.
> In order to increase the memory usage (reduce unnecessary spilling) also
> reduce the risk of OOM, we should manage the memory in a cooperative way, it
> means all the memory consume should be also responsive to release memory
> (spilling) upon others' requests.
> The requests of memory could be different, hard requirement (will crash if
> not allocated) or soft requirement (worse performance if not allocated). Also
> the costs of spilling are also different. We could introduce some kind of
> priority to make them work together better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
