Which version of Spark is this in, Koert? There might have been some fixes more recently for it.
Matei On Mar 5, 2014, at 5:26 PM, Koert Kuipers <[email protected]> wrote: > Sorry I meant to say: seems the issue is shared RDDs between a job that got > cancelled and a later job. > > However even disregarding that I have the other issue that the active task of > the cancelled job hangs around forever, not doing anything.... > > On Mar 5, 2014 7:29 PM, "Koert Kuipers" <[email protected]> wrote: > yes jobs on RDDs that were not part of the cancelled job work fine. > > so it seems the issue is the cached RDDs that are ahred between the cancelled > job and the jobs after that. > > > On Wed, Mar 5, 2014 at 7:15 PM, Koert Kuipers <[email protected]> wrote: > well, the new jobs use existing RDDs that were also used in the jon that got > killed. > > let me confirm that new jobs that use completely different RDDs do not get > killed. > > > > On Wed, Mar 5, 2014 at 7:00 PM, Mayur Rustagi <[email protected]> wrote: > Quite unlikely as jobid are given in an incremental fashion, so your future > jobid are not likely to be killed if your groupid is not repeated.I guess the > issue is something else. > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > > On Wed, Mar 5, 2014 at 3:50 PM, Koert Kuipers <[email protected]> wrote: > i did that. my next job gets a random new group job id (a uuid). however that > doesnt seem to stop the job from getting sucked into the cancellation it seems > > > On Wed, Mar 5, 2014 at 6:47 PM, Mayur Rustagi <[email protected]> wrote: > You can randomize job groups as well. to secure yourself against termination. > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > > On Wed, Mar 5, 2014 at 3:42 PM, Koert Kuipers <[email protected]> wrote: > got it. seems like i better stay away from this feature for now.. > > > On Wed, Mar 5, 2014 at 5:55 PM, Mayur Rustagi <[email protected]> wrote: > One issue is that job cancellation is posted on eventloop. So its possible > that subsequent jobs submitted to job queue may beat the job cancellation > event & hence the job cancellation event may end up closing them too. > So there's definitely a race condition you are risking even if not running > into. > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > > On Wed, Mar 5, 2014 at 2:40 PM, Koert Kuipers <[email protected]> wrote: > SparkContext.cancelJobGroup > > > On Wed, Mar 5, 2014 at 5:32 PM, Mayur Rustagi <[email protected]> wrote: > How do you cancel the job. Which API do you use? > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > > On Wed, Mar 5, 2014 at 2:29 PM, Koert Kuipers <[email protected]> wrote: > i also noticed that jobs (with a new JobGroupId) which i run after this use > which use the same RDDs get very confused. i see lots of cancelled stages and > retries that go on forever. > > > On Tue, Mar 4, 2014 at 5:02 PM, Koert Kuipers <[email protected]> wrote: > i have a running job that i cancel while keeping the spark context alive. > > at the time of cancellation the active stage is 14. > > i see in logs: > 2014/03/04 16:43:19 INFO scheduler.DAGScheduler: Asked to cancel job group > 3a25db23-2e39-4497-b7ab-b26b2a976f9c > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Cancelling stage 10 > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Cancelling stage 14 > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Stage 14 was cancelled > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 14.0 > from pool x > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Cancelling stage 13 > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Cancelling stage 12 > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Cancelling stage 11 > 2014/03/04 16:43:19 INFO scheduler.TaskSchedulerImpl: Cancelling stage 15 > > so far it all looks good. then i get a lot of messages like this: > 2014/03/04 16:43:20 INFO scheduler.TaskSchedulerImpl: Ignoring update with > state FINISHED from TID 883 because its task set is gone > 2014/03/04 16:43:24 INFO scheduler.TaskSchedulerImpl: Ignoring update with > state KILLED from TID 888 because its task set is gone > > after this stage 14 hangs around in active stages, without any sign of > progress or cancellation. it just sits there forever, stuck. looking at the > logs of the executors confirms this. they task seem to be still running, but > nothing is happening. for example (by the time i look at this its 4:58 so > this tasks hasnt done anything in 15 mins): > > 14/03/04 16:43:16 INFO Executor: Serialized size of result for 943 is 1007 > 14/03/04 16:43:16 INFO Executor: Sending result for 943 directly to driver > 14/03/04 16:43:16 INFO Executor: Finished task ID 943 > 14/03/04 16:43:16 INFO Executor: Serialized size of result for 945 is 1007 > 14/03/04 16:43:16 INFO Executor: Sending result for 945 directly to driver > 14/03/04 16:43:16 INFO Executor: Finished task ID 945 > 14/03/04 16:43:19 INFO BlockManager: Removing RDD 66 > > not sure what to make of this. any suggestions? best, koert > > > > > > > > > >
