Kazuyuki Tanimura created SPARK-53048:
-----------------------------------------
Summary: Include PySpark exception reasons in SparkUserAppException
Key: SPARK-53048
URL: https://issues.apache.org/jira/browse/SPARK-53048
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0, 4.1.0
Reporter: Kazuyuki Tanimura
Currently the exit code of Python process failure is passed to
SparkUserAppException.
In particular this piece of code
[https://github.com/apache/spark/blob/3bf3696461ab18e6b2507f3f1a2729568e4a6832/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L127]
```
try {
val process = builder.start()
new RedirectThread(process.getInputStream, System.out, "redirect
output").start()
val exitCode = process.waitFor()
if (exitCode != 0) {
throw new SparkUserAppException(exitCode)
}
} finally {
gatewayServer.foreach(_.shutdown())
}
```
Although logger can be set for PySpark, it would be more user friendly if we
can include more explicit reasons or Python stacktrace, so that users do not
have to look into the other part of the logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
