[
https://issues.apache.org/jira/browse/SPARK-53048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-53048:
----------------------------------
Affects Version/s: (was: 4.0.0)
> Include PySpark exception reasons in SparkUserAppException
> ----------------------------------------------------------
>
> Key: SPARK-53048
> URL: https://issues.apache.org/jira/browse/SPARK-53048
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Kazuyuki Tanimura
> Priority: Major
>
> Currently the exit code of Python process failure is passed to
> SparkUserAppException.
> In particular this piece of code
> [https://github.com/apache/spark/blob/3bf3696461ab18e6b2507f3f1a2729568e4a6832/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L127]
> ```
> try {
> val process = builder.start()
>
> new RedirectThread(process.getInputStream, System.out, "redirect
> output").start()
>
> val exitCode = process.waitFor()
> if (exitCode != 0) {
> throw new SparkUserAppException(exitCode)
> }
> } finally {
> gatewayServer.foreach(_.shutdown())
> }
> ```
> Although logger can be set for PySpark, it would be more user friendly if we
> can include more explicit reasons or Python stacktrace, so that users do not
> have to look into the other part of the logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
