Github user sundeepn commented on the pull request:
https://github.com/apache/spark/pull/246#issuecomment-39125321
@kayousterhout I have thought about adding cancelStage to
SparkContext/DAGScheduler. My earlier take was that the Job level Info is
useful in the UI as well. Currently, its not shown/used in the UI, but lots of
our users get confused looking at the stage level UI showing their query
multiple times. It will be good to expose the Job ID in the stage table as
well.
Having said that, I do not see any other way of doing this. We will have to
move this to the DAGScheduler as a cancelStage. If two jobs can share stages,
then any handling from the UI side can have a race condition in the cancel work
flow and it will be a lot cleaner to handle upstream. I will submit a revision
shortly.
@andrewor14, What do you think about adding a jobId column to the Stage
table?
Thanks Mark for the pointer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
