Driver timing out laggards seems like a reasonable way of handling
laggards. Are there any challenges because of which driver does not do it
today? Is there a JIRA for this? I couldn't find one.
On Tue, Sep 15, 2015 at 12:07 PM, Akhil Das
wrote:
> As of now i
As of now i think its a no. Not sure if its a naive approach, but yes you
can have a separate program to keep an eye in the webui (possibly parsing
the content) and make it trigger the kill task/job once it detects a lag.
(Again you will have to figure out the correct numbers before killing any
Thanks, Mark, will look into that...
On Tue, Sep 15, 2015 at 12:33 PM, Mark Hamstra
wrote:
> There is the Async API (
> https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala),
> which makes use of FutureAction
There is the Async API (
https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala),
which makes use of FutureAction (
https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/FutureAction.scala).
You could
Is there a way in Spark to automatically terminate laggard "stage's", ones
that appear to be hanging? In other words, is there a timeout for
processing of a given RDD?
In the Spark GUI, I see the "kill" function for a given Stage under
'Details for Job <...>".
Is there something in Spark that