There is a parameter spark.speculation that is turned off by default. Look at the configuration doc: http://spark.apache.org/docs/latest/configuration.html
From: Pramod Biligiri <[email protected]<mailto:[email protected]>> Date: Monday, September 15, 2014 at 3:30 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Does Spark always wait for stragglers to finish running? Hi, I'm running Spark tasks with speculation enabled. I'm noticing that Spark seems to wait in a given stage for all stragglers to finish, even though the speculated alternative might have finished sooner. Is that correct? Is there a way to indicate to Spark not to wait for stragglers to finish? Thanks, Pramod -- http://twitter.com/pramodbiligiri
