It can be used, and is used in user code, but it isn't always as straightforward as you might think. This is mostly because a Job often isn't a Job -- or rather it is more than one Job. There are several RDD transformations that aren't lazy, so they end up launching "hidden" Jobs that you may not anticipate and may expect to be canceled (but won't be) by a cancelJob() called on a later action on that transformed RDD. It is also possible for a single DataFrame or Spark SQL query to result in more than one running Job. The upshot of all of this is that getting cancelJob() to work as most users would expect all the time is non-trivial, and most of the time using a jobGroup is a better way to capture what may be more than one Job that the user is thinking of as a single Job.
On Wed, Dec 16, 2015 at 5:34 AM, Sean Owen <so...@cloudera.com> wrote: > It does look like it's not actually used. It may simply be there for > completeness, to match cancelStage and cancelJobGroup, which are used. > I also don't know of a good reason there's no way to kill a whole job. > > On Wed, Dec 16, 2015 at 1:15 PM, Jacek Laskowski <ja...@japila.pl> wrote: > > Hi, > > > > While reviewing Spark code I came across SparkContext.cancelJob. I > > found no part of Spark using it. Is this a leftover after some > > refactoring? Why is this part of sc? > > > > The reason I'm asking is another question I'm having after having > > learnt about killing a stage in webUI. I noticed there is a way to > > kill/cancel stages, but no corresponding feature to kill/cancel jobs. > > Why? Is there a JIRA ticket to have it some day perhaps? > > > > Pozdrawiam, > > Jacek > > > > -- > > Jacek Laskowski | https://medium.com/@jaceklaskowski/ > > Mastering Apache Spark > > ==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ > > Follow me at https://twitter.com/jaceklaskowski > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
