[ https://issues.apache.org/jira/browse/SPARK-33339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-33339. ---------------------------------- Fix Version/s: 3.1.0 3.0.2 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/30248 > Pyspark application will hang due to non Exception > -------------------------------------------------- > > Key: SPARK-33339 > URL: https://issues.apache.org/jira/browse/SPARK-33339 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.4.5, 3.0.0, 3.0.1 > Reporter: lrz > Priority: Major > Fix For: 3.0.2, 3.1.0 > > > When a system.exit exception occurs during the process, the python worker > exits abnormally, and then the executor task is still waiting for the worker > for reading from socket, causing it to hang. > The system.exit exception may be caused by the user's error code, but spark > should at least throw an error to remind the user, not get stuck > we can run a simple test to reproduce this case: > ``` > from pyspark.sql import SparkSession > def err(line): > raise SystemExit > spark = SparkSession.builder.appName("test").getOrCreate() > spark.sparkContext.parallelize(range(1,2), 2).map(err).collect() > spark.stop() > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org