Andrew Or created SPARK-3493: -------------------------------- Summary: Unrolling behavior is too aggressive in dropping blocks Key: SPARK-3493 URL: https://issues.apache.org/jira/browse/SPARK-3493 Project: Spark Issue Type: Improvement Components: Block Manager Affects Versions: 1.1.0 Reporter: Andrew Or Assignee: Andrew Or Priority: Critical
When unrolling a new block, we (almost) always call `ensureFreeSpace` with a fixed fraction (spark.storage.unrollFraction) of the block manager's total memory minus a small delta. This means we're essentially reserving this chunk of memory in advance because we allocate all of this space even if the new block is very small. We need to call `ensureFreeSpace` incrementally and more cautiously to avoid unnecessarily dropping blocks where possible. Thanks to [~tdas] for originally reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org