[
https://issues.apache.org/jira/browse/SPARK-48900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-48900.
---------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 47361
[https://github.com/apache/spark/pull/47361]
> Add `reason` string to all job / stage / job group cancellation calls
> ---------------------------------------------------------------------
>
> Key: SPARK-48900
> URL: https://issues.apache.org/jira/browse/SPARK-48900
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Mingkang Li
> Assignee: Mingkang Li
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Today it is difficult to determine _why_ a job, stage, or job group was
> canceled. We should leverage existing Spark functionality to provide a
> {{reason}} string explaining the cancellation cause, and should add new APIs
> to let us provide this reason when canceling job groups.
> {*}Details{*}:
> * Since SPARK-19549 Allow providing reasons for stage/job cancelling - ASF
> JIRA (Spark 2.20), Spark’s {{cancelJob}} and {{cancelStage}} methods accept
> an optional {{reason: String}} that is added to logging output and
> user-facing error messages when jobs or stages are canceled. In our internal
> calls to these methods, we should always supply a reason. For example, we
> should set an appropriate reason when the “kill” links are clicked in the
> Spark UI (see
> [code|https://github.com/apache/spark/blob/b14c1f036f8f394ad1903998128c05d04dd584a9/core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala#L54C1-L55]).
> * Other APIs currently lack a {{reason}} field. For example,
> {{cancelJobGroup}} and {{cancelJobsWithTag}} don’t provide any way to specify
> a reason, so we only see generic logs like “asked to cancel job group <group
> name>”. We should add an ability to pass in a group cancellation reason and
> thread that through into the scheduler’s logging and job failure reasons.
> This feature can be implemented in two PRs:
> # Modify the current {{SparkContext}} and its downstream APIs to add the
> {{reason}} string, such as {{cancelJobGroup}} and {{cancelJobsWithTag}}
> 2. Add reasons for all internal calls to these methods
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
