[ https://issues.apache.org/jira/browse/SPARK-48900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-48900: ----------------------------------- Assignee: Mingkang Li > Add `reason` string to all job / stage / job group cancellation calls > --------------------------------------------------------------------- > > Key: SPARK-48900 > URL: https://issues.apache.org/jira/browse/SPARK-48900 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Affects Versions: 4.0.0 > Reporter: Mingkang Li > Assignee: Mingkang Li > Priority: Major > Labels: pull-request-available > > Today it is difficult to determine _why_ a job, stage, or job group was > canceled. We should leverage existing Spark functionality to provide a > {{reason}} string explaining the cancellation cause, and should add new APIs > to let us provide this reason when canceling job groups. > {*}Details{*}: > * Since SPARK-19549 Allow providing reasons for stage/job cancelling - ASF > JIRA (Spark 2.20), Spark’s {{cancelJob}} and {{cancelStage}} methods accept > an optional {{reason: String}} that is added to logging output and > user-facing error messages when jobs or stages are canceled. In our internal > calls to these methods, we should always supply a reason. For example, we > should set an appropriate reason when the “kill” links are clicked in the > Spark UI (see > [code|https://github.com/apache/spark/blob/b14c1f036f8f394ad1903998128c05d04dd584a9/core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala#L54C1-L55]). > * Other APIs currently lack a {{reason}} field. For example, > {{cancelJobGroup}} and {{cancelJobsWithTag}} don’t provide any way to specify > a reason, so we only see generic logs like “asked to cancel job group <group > name>”. We should add an ability to pass in a group cancellation reason and > thread that through into the scheduler’s logging and job failure reasons. > This feature can be implemented in two PRs: > # Modify the current {{SparkContext}} and its downstream APIs to add the > {{reason}} string, such as {{cancelJobGroup}} and {{cancelJobsWithTag}} > 2. Add reasons for all internal calls to these methods -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org