lrz created SPARK-33339:
---------------------------
Summary: at pyspark application, worker don't catch BaseException,
then once worker call system.exit because of some error, the application will
hangup, and will not throw any exception, then the fail cause is not easy to
find.
Key: SPARK-33339
URL: https://issues.apache.org/jira/browse/SPARK-33339
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.0.1, 3.0.0, 2.4.5
Reporter: lrz
for example, run `spark-submit --master yarn-client test.py`, this command
will hangup without any information. The test.py content:
```
from pyspark.sql import SparkSession
def err(line):
raise SystemExit
spark = SparkSession.builder.appName("test").getOrCreate()
spark.sparkContext.parallelize(range(1,2), 2).map(err).collect()
spark.stop
```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
