[jira] (SPARK-19371) Cannot spread cached partitions evenly across executors

[Sean Owen (JIRA)]

Title: Message Title

Sean Owen commented on SPARK-19371   Re: Cannot spread cached partitions evenly across executors In your case you want to cache a copy of the data in memory, which makes the locality of the original data not relevant (good, because you can't always get locality even on HDFS). Once in memory, being unbalanced across executors isn't a big deal, because (in theory, but mostly true in practice) one execution slot is as good as another, because it's a core operating on local memory. I do believe Spark already prefers spreading tasks across executors, all else equal. A shuffle could be the right thing in front of this operation, because there's going to be some copying already, so the extra cost isn't high. Coupled with not waiting for locality on the initial read I think this is indeed the right approach in Spark now. Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)