[ https://issues.apache.org/jira/browse/SPARK-33339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-33339: ---------------------------------- Fix Version/s: 2.4.8 > Pyspark application will hang due to non Exception > -------------------------------------------------- > > Key: SPARK-33339 > URL: https://issues.apache.org/jira/browse/SPARK-33339 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.4.5, 3.0.0, 3.0.1 > Reporter: lrz > Assignee: lrz > Priority: Major > Fix For: 2.4.8, 3.0.2, 3.1.0 > > > When a system.exit exception occurs during the process, the python worker > exits abnormally, and then the executor task is still waiting for the worker > for reading from socket, causing it to hang. > The system.exit exception may be caused by the user's error code, but spark > should at least throw an error to remind the user, not get stuck > we can run a simple test to reproduce this case: > {code:python} > from pyspark.sql import SparkSession > def err(line): > raise SystemExit > spark = SparkSession.builder.appName("test").getOrCreate() > spark.sparkContext.parallelize(range(1,2), 2).map(err).collect() > spark.stop() > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org