GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/7585
[SPARK-9144] Remove DAGScheduler.runLocallyWithinThread and
spark.localExecution.enabled
Spark has an option called spark.localExecution.enabled; according to the
docs:
```
Enables Spark to run certain jobs, such as first() or take() on the driver,
without sending tasks to the cluster. This can make certain jobs execute very
quickly, but may require shipping a whole partition of data to the driver.
```
This feature ends up adding quite a bit of complexity to DAGScheduler,
especially in the runLocallyWithinThread method, but as far as I know nobody
uses this feature (I searched the mailing list and haven't seen any recent
mentions of the configuration nor stacktraces including the runLocally method).
As a step towards scheduler complexity reduction, I propose that we remove this
feature and all code related to it for Spark 1.5.
This pull request simply brings #7484 up to date.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark remove-local-exec
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7585.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7585
----
commit ffa8c9b6091559fb6f69d89d000eaa1b17dedef8
Author: Josh Rosen <joshro...@databricks.com>
Date: 2015-07-17T22:10:39Z
Remove documentation for configuration
commit 8975d965215d02e0ed7b963325ece4a4645bb870
Author: Josh Rosen <joshro...@databricks.com>
Date: 2015-07-17T22:16:21Z
Remove local execution tests.
commit b0835dc09dc6d7df1856fa96667ae56749ef8ce9
Author: Josh Rosen <joshro...@databricks.com>
Date: 2015-07-17T22:21:07Z
Remove local execution code in DAGScheduler
commit eec39fa1d5d8f9f5bdfaf7b4907f2fe20b635c8f
Author: Josh Rosen <joshro...@databricks.com>
Date: 2015-07-17T22:53:48Z
Remove allowLocal(); deprecate user-facing uses of it.
commit 1d9739ab232ff04b85cff8cec348d97b829965e3
Author: Reynold Xin <r...@databricks.com>
Date: 2015-07-22T06:12:35Z
Merge pull request #7484 from JoshRosen/remove-localexecution
[SPARK-9144] Remove DAGScheduler.runLocallyWithinThread and
spark.localExecution.enabled
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
