[
https://issues.apache.org/jira/browse/SPARK-53048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuyuki Tanimura updated SPARK-53048:
--------------------------------------
Description:
Currently the exit code of Python process failure is passed to
SparkUserAppException.
In particular this piece of code
[https://github.com/apache/spark/blob/3bf3696461ab18e6b2507f3f1a2729568e4a6832/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L127]
```
try {
val process = builder.start()
new RedirectThread(process.getInputStream, System.out, "redirect
output").start()
val exitCode = process.waitFor()
if (exitCode != 0)
{ throw new SparkUserAppException(exitCode) }
} finally
{ gatewayServer.foreach(_.shutdown()) }
```
Although logger can be set for PySpark, it would be more user friendly if we
can include more explicit reasons or Python stacktrace to
SparkUserAppException, so that users do not have to look into the other part of
the logs
was:
Currently the exit code of Python process failure is passed to
SparkUserAppException.
In particular this piece of code
[https://github.com/apache/spark/blob/3bf3696461ab18e6b2507f3f1a2729568e4a6832/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L127]
```
try {
val process = builder.start()
new RedirectThread(process.getInputStream, System.out, "redirect
output").start()
val exitCode = process.waitFor()
if (exitCode != 0) {
throw new SparkUserAppException(exitCode)
}
} finally {
gatewayServer.foreach(_.shutdown())
}
```
Although logger can be set for PySpark, it would be more user friendly if we
can include more explicit reasons or Python stacktrace, so that users do not
have to look into the other part of the logs
> Include PySpark exception reasons in SparkUserAppException
> ----------------------------------------------------------
>
> Key: SPARK-53048
> URL: https://issues.apache.org/jira/browse/SPARK-53048
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Kazuyuki Tanimura
> Priority: Major
>
> Currently the exit code of Python process failure is passed to
> SparkUserAppException.
> In particular this piece of code
> [https://github.com/apache/spark/blob/3bf3696461ab18e6b2507f3f1a2729568e4a6832/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L127]
> ```
> try {
> val process = builder.start()
>
> new RedirectThread(process.getInputStream, System.out, "redirect
> output").start()
>
> val exitCode = process.waitFor()
> if (exitCode != 0)
> { throw new SparkUserAppException(exitCode) }
> } finally
> { gatewayServer.foreach(_.shutdown()) }
> ```
> Although logger can be set for PySpark, it would be more user friendly if we
> can include more explicit reasons or Python stacktrace to
> SparkUserAppException, so that users do not have to look into the other part
> of the logs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
