As of now i think its a no. Not sure if its a naive approach, but yes you can have a separate program to keep an eye in the webui (possibly parsing the content) and make it trigger the kill task/job once it detects a lag. (Again you will have to figure out the correct numbers before killing any job)
Thanks Best Regards On Mon, Sep 14, 2015 at 10:40 PM, Dmitry Goldenberg < [email protected]> wrote: > Is there a way in Spark to automatically terminate laggard "stage's", ones > that appear to be hanging? In other words, is there a timeout for > processing of a given RDD? > > In the Spark GUI, I see the "kill" function for a given Stage under > 'Details for Job <...>". > > Is there something in Spark that would identify and kill laggards > proactively? > > Thanks. >
