GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/7385
[SPARK-9026] Refactor SimpleFutureAction.onComplete to not launch separate thread for every callback This commit refactors SimpleFutureAction so that registering a callback with `onComplete` does not immediately tie up a thread in the provided execution context. As @zsxwing noticed in https://github.com/apache/spark/pull/7276#issuecomment-121097747, the existing implementation of SimpleFutureAction.onComplete creates a separate thread to wrap the blocking `awaitResult()` callback: ```scala override def onComplete[U](func: (Try[T]) => U)(implicit executor: ExecutionContext) { executor.execute(new Runnable { override def run() { func(awaitResult()) } }) } ``` This PR addresses this issue by adding a future to JobWaiter and using that future in the implementation of SimpleFutureAction's future methods. The idea of adding a future to JobWaiter is borrowed from #7276. This patch was pair-programmed with @tdas. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark simplefutureaction-refactoring Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7385.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7385 ---- commit 11c745095bbaa382a15e6ad1ee1afd2e933e4da8 Author: Josh Rosen <joshro...@databricks.com> Date: 2015-07-14T02:37:38Z Refactor SimpleFutureAction to not block threads for every onComplete callback. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org