[
https://issues.apache.org/jira/browse/SPARK-33339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lrz updated SPARK-33339:
------------------------
Description:
at pyspark application, worker don't catch BaseException, then once worker call
system.exit because of some error, the application will hangup, and will not
throw any exception, then the fail cause is not easy to find.
for example, run `spark-submit --master yarn-client test.py`, this command
will hangup without any information. The test.py content:
```
from pyspark.sql import SparkSession
def err(line):
raise SystemExit
spark = SparkSession.builder.appName("test").getOrCreate()
spark.sparkContext.parallelize(range(1,2), 2).map(err).collect()
spark.stop()
```
was:
for example, run `spark-submit --master yarn-client test.py`, this command
will hangup without any information. The test.py content:
```
from pyspark.sql import SparkSession
def err(line):
raise SystemExit
spark = SparkSession.builder.appName("test").getOrCreate()
spark.sparkContext.parallelize(range(1,2), 2).map(err).collect()
spark.stop()
```
Summary: pyspark application maybe hangup because of worker exit (was:
at pyspark application, worker don't catch BaseException, then once worker call
system.exit because of some error, the application will hangup, and will not
throw any exception, then the fail cause is not easy to find.)
> pyspark application maybe hangup because of worker exit
> -------------------------------------------------------
>
> Key: SPARK-33339
> URL: https://issues.apache.org/jira/browse/SPARK-33339
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.5, 3.0.0, 3.0.1
> Reporter: lrz
> Priority: Major
>
> at pyspark application, worker don't catch BaseException, then once worker
> call system.exit because of some error, the application will hangup, and will
> not throw any exception, then the fail cause is not easy to find.
> for example, run `spark-submit --master yarn-client test.py`, this command
> will hangup without any information. The test.py content:
> ```
> from pyspark.sql import SparkSession
> def err(line):
> raise SystemExit
> spark = SparkSession.builder.appName("test").getOrCreate()
> spark.sparkContext.parallelize(range(1,2), 2).map(err).collect()
> spark.stop()
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
