Ganesha S created SPARK-58617:
---------------------------------
Summary: Allow opting out of the context-wide cancelAllJobs on
SIGINT
Key: SPARK-58617
URL: https://issues.apache.org/jira/browse/SPARK-58617
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.3.0
Reporter: Ganesha S
{\{SparkContext._do_init}} installs a SIGINT handler that calls
\{{cancelAllJobs()}} before raising \{{KeyboardInterrupt}}, so that Ctrl-C in
the pyspark shell aborts the running job instead of forcing the user to restart
the shell (SPARK-8170).
That design assumes the interrupted process is the only user of the
\{{SparkContext}}. When several independent Python processes share one context,
the assumption does not hold: \{{cancelAllJobs()}} is context-wide rather than
scoped to the interrupted command, so a SIGINT delivered to one process also
aborts healthy in-flight jobs belonging to the others. Those jobs fail with:
{noformat}
[SPARK_JOB_CANCELLED] Job N cancelled as part of cancellation of all jobs
{noformat}
which reads as a failure of the victim job rather than collateral of an
unrelated process's interrupt. An embedder that has its own per-command
cancellation currently has no way to keep the \{{KeyboardInterrupt}} while
declining the context-wide cancel: the two are welded together in the handler.
Proposal: gate the cancel on a new \{{spark.python.cancelAllJobsOnSigint}},
defaulting to \{{true}} so existing behavior is unchanged. The handler is still
installed in both cases, so \{{KeyboardInterrupt}} is raised either way and
interrupting user code is unaffected. Setting it to \{{false}} yields SIGINT
semantics without the context-wide cancel.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
