It's true that it does not send a kill command right now -- we should probably add that. This code was written before tasks were killable AFAIK. However, the *job* should still finish while a speculative task is running as far as I know, and it will just leave that task behind.
Matei On September 15, 2014 at 4:51:59 PM, Pramod Biligiri (pramodbilig...@gmail.com) wrote: I'm already running with speculation set to true and the speculated tasks are launching, but the issue I'm observing is that Spark does not kill the long running task even if the shorter alternative has finished successfully. Therefore the overall turnaround time is still the same as without speculation. Pramod On Mon, Sep 15, 2014 at 4:22 PM, Du Li <l...@yahoo-inc.com> wrote: There is a parameter spark.speculation that is turned off by default. Look at the configuration doc: http://spark.apache.org/docs/latest/configuration.html From: Pramod Biligiri <pramodbilig...@gmail.com> Date: Monday, September 15, 2014 at 3:30 PM To: "user@spark.apache.org" <user@spark.apache.org> Subject: Does Spark always wait for stragglers to finish running? Hi, I'm running Spark tasks with speculation enabled. I'm noticing that Spark seems to wait in a given stage for all stragglers to finish, even though the speculated alternative might have finished sooner. Is that correct? Is there a way to indicate to Spark not to wait for stragglers to finish? Thanks, Pramod -- http://twitter.com/pramodbiligiri
