[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
DjvuLee updated SPARK-21082: ---------------------------- Affects Version/s: (was: 2.2.1) 2.3.0 > Consider Executor's memory usage when scheduling task > ------------------------------------------------------ > > Key: SPARK-21082 > URL: https://issues.apache.org/jira/browse/SPARK-21082 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Spark Core > Affects Versions: 2.3.0 > Reporter: DjvuLee > > Spark Scheduler do not consider the memory usage during dispatch tasks, this > can lead to Executor OOM if the RDD is cached sometimes, because Spark can > not estimate the memory usage well enough(especially when the RDD type is not > flatten), scheduler may dispatch so many tasks on one Executor. > We can offer a configuration for user to decide whether scheduler will > consider the memory usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org