Thanks, Mark, will look into that... On Tue, Sep 15, 2015 at 12:33 PM, Mark Hamstra <m...@clearstorydata.com> wrote:
> There is the Async API ( > https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala), > which makes use of FutureAction ( > https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/FutureAction.scala). > You could also wrap up your Jobs in Futures on your own. > > On Mon, Sep 14, 2015 at 11:37 PM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> As of now i think its a no. Not sure if its a naive approach, but yes you >> can have a separate program to keep an eye in the webui (possibly parsing >> the content) and make it trigger the kill task/job once it detects a lag. >> (Again you will have to figure out the correct numbers before killing any >> job) >> >> Thanks >> Best Regards >> >> On Mon, Sep 14, 2015 at 10:40 PM, Dmitry Goldenberg < >> dgoldenberg...@gmail.com> wrote: >> >>> Is there a way in Spark to automatically terminate laggard "stage's", >>> ones that appear to be hanging? In other words, is there a timeout for >>> processing of a given RDD? >>> >>> In the Spark GUI, I see the "kill" function for a given Stage under >>> 'Details for Job <...>". >>> >>> Is there something in Spark that would identify and kill laggards >>> proactively? >>> >>> Thanks. >>> >> >> >
