Maciej Szymkiewicz created SPARK-38073: ------------------------------------------
Summary: NameError: name 'sc' is not defined when running driver with IPyhon and Pyhon > 3.7 Key: SPARK-38073 URL: https://issues.apache.org/jira/browse/SPARK-38073 Project: Spark Issue Type: Improvement Components: PySpark, Spark Shell Affects Versions: 3.2.0, 3.3.0 Reporter: Maciej Szymkiewicz When {{PYSPARK_DRIVER_PYTHON=$(which ipython) bin/pyspark}} is executed with Python >= 3.8, function registered wiht atexit seems to be executed in different scope than in Python 3.7. It result in {{NameError: name 'sc' is not defined}} on exit: {code:python} Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 3.3.0-SNAPSHOT /_/ Using Python version 3.8.12 (default, Oct 12 2021 21:57:06) Spark context Web UI available at http://192.168.0.198:4040 Spark context available as 'sc' (master = local[*], app id = local-1643555855409). SparkSession available as 'spark'. In [1]: Do you really want to exit ([y]/n)? y Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/path/to/spark/python/pyspark/shell.py", line 49, in <lambda> atexit.register(lambda: sc.stop()) NameError: name 'sc' is not defined {code} This could be easily fixed by capturing `sc` instance {code:none} diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py index f0c487877a..4164e3ab0c 100644 --- a/python/pyspark/shell.py +++ b/python/pyspark/shell.py @@ -46,7 +46,7 @@ except Exception: sc = spark.sparkContext sql = spark.sql -atexit.register(lambda: sc.stop()) +atexit.register((lambda sc: lambda: sc.stop())(sc)) # for compatibility sqlContext = spark._wrapped {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org