[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated SPARK-3466: --------------------------------- Priority: Critical (was: Major) > Limit size of results that a driver collects for each action > ------------------------------------------------------------ > > Key: SPARK-3466 > URL: https://issues.apache.org/jira/browse/SPARK-3466 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Reporter: Matei Zaharia > Assignee: Davies Liu > Priority: Critical > > Right now, operations like {{collect()}} and {{take()}} can crash the driver > with an OOM if they bring back too many data. We should add a > {{spark.driver.maxResultSize}} setting (or something like that) that will > make the driver abort a job if its result is too big. We can set it to some > fraction of the driver's memory by default, or to something like 100 MB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org