GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1452
[SPARK-2521] Broadcast RDD object once per TaskSet (instead of sending it for every task). This should substantially reduces task size for large jobs (with large closures). It should improve task dispatching and scheduler throughput. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark broadcast-task Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1452.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1452 ---- commit 73b2783fef785941fc966ad32f2fd987b12447ae Author: Reynold Xin <r...@apache.org> Date: 2014-07-16T23:34:34Z [SPARK-2534] Avoid pulling in the entire RDD in groupByKey. commit 2f7fc5f00b6bbb3970224119d4380f0117fcb5be Author: Reynold Xin <r...@apache.org> Date: 2014-07-16T23:53:35Z [SPARK-2521] Broadcast RDD object once per TaskSet (instead of sending it for every task). commit 9a4a770f0350a863f9096d4486aa124af9a07d6a Author: Reynold Xin <r...@apache.org> Date: 2014-07-16T23:56:55Z Explain why broadcasting serialized copy of the task. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---