[ https://issues.apache.org/jira/browse/SPARK-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or resolved SPARK-11296. ------------------------------- Resolution: Duplicate > FutureAction.onComplete consumes a thread while waiting for job > --------------------------------------------------------------- > > Key: SPARK-11296 > URL: https://issues.apache.org/jira/browse/SPARK-11296 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Spark Core > Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1 > Reporter: Richard W. Eggert II > Labels: performance > Original Estimate: 24h > Remaining Estimate: 24h > > The implementations of the onComplete callback (and, by extension, the > callbacks based on it) of FutureAction (SimpleFutureAction and > ComplexFutureAction) each consume a thread that blocks waiting for the > underlying job(s) to complete. This is unnecessary and is likely to be very > surprising to anyone accustomed to the usual behavior of Scala Futures (i.e., > that callbacks only occupy threads AFTER the underlying task has completed), > and it runs the risk of causing thread starvation for any processes utilizing > the same thread pool (especially if scala.concurrent.ExecutionContext.global > is used). The implementations can and should be reworked to operate in a > non-blocking manner, by modifying the underlying JobWaiter class to expose a > non-blocking API, so that the onComplete (as well as onSuccess and onFailure) > callbacks only utilize threads while they are actually executing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org