[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048653#comment-16048653
]
DjvuLee commented on SPARK-21082:
---------------------------------
[~srowen] This situation occurred when the partition number is larger than the
CPU core.
Consider there are 1000 partition and 100 CPU core, we want cache RDD among all
the Executors.
If one Executor executes tasks fast at first time, then the scheduler will
dispatch more tasks to it,
so after all the tasks is scheduled, some Executors will used all the storage
memory, but some Executors just use few memory,
Executors which used more memory may not cache all the RDD partition scheduled
on it, because there is no more memory for some tasks.
Under this situation, we can not cache all the partition even we have enough
memory.
What's more, if some Executors occurred OOM during following compute, the
scheduler may dispatch tasks to Executor which have no more storage memory, and
sometimes can lead to more and more OOM if Spark can not estimate the memory.
But if the scheduler try to schedule tasks to Executors which own more free
memory can ease this situation.
Maybe we can use the `coalesce` to decrease the partition number, but this is
not good enough for speculating.
> Consider Executor's memory usage when scheduling task
> ------------------------------------------------------
>
> Key: SPARK-21082
> URL: https://issues.apache.org/jira/browse/SPARK-21082
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
> Affects Versions: 2.2.1
> Reporter: DjvuLee
>
> Spark Scheduler do not consider the memory usage during dispatch tasks, this
> can lead to Executor OOM if the RDD is cached sometimes, because Spark can
> not estimate the memory usage enough well(especially when the RDD type is not
> flatten), scheduler may dispatch so many tasks on one Executor.
> We can offer a configuration for user to decide whether scheduler will
> consider the memory usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
