Mingyu Kim created SPARK-5915:
---------------------------------
Summary: Spillable should check every N bytes rather than every 32
elements
Key: SPARK-5915
URL: https://issues.apache.org/jira/browse/SPARK-5915
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Mingyu Kim
Spillable currently checks for spill every 32 elements. However, this puts it
at a risk of OOM if each element is large enough. A better alternative is to
check every N bytes accumulated.
N should be decided to a reasonable number via proper testing.
This is a follow-up of SPARK-4808, and was discussed originally in
https://github.com/apache/spark/pull/4420.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
