[jira] (SPARK-19371) Cannot spread cached partitions evenly across executors

[Thunder Stumpges (JIRA)]

Title: Message Title

Thunder Stumpges edited a comment on SPARK-19371   Re: Cannot spread cached partitions evenly across executors {quote} ... one execution slot is as good as another, because it's a core operating on local memory. I do believe Spark already prefers spreading tasks across executors, all else equal. {quote} So, this does not seem to be what we're seeing. Here's output from our executors tab after running a while. Notice the executors with more RDD partitions have proportionally more tasks executed. ||Executor ID||RDD Blocks||Storage Memory||Disk Used||Active Tasks||Failed Tasks||Complete Tasks||Total Tasks || | 1|8|91.2 MB / 2.7 GB|0.0 B|0|0|440804|440804 | | 10|30|342.0 MB / 2.7 GB|0.0 B|0|0|722964|722964 | | 11|8|91.2 MB / 2.7 GB|0.0 B|0|0|444454|444454 | | 12|27|307.8 MB / 2.7 GB|0.0 B|0|0|780879|780879 | With RDD partitions cached, wouldn't the scheduler prefer to run the task on the executor containing the blocks? Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)