[ https://issues.apache.org/jira/browse/SPARK-38912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Furcy Pin updated SPARK-38912: ------------------------------ Description: *Context* The proper bugfix of SPARK-38870 would be to use the @classmethod + @property decorators on the SparkSession.builder attribute to make it return a new builder every time like in Scala. However, @classmethod + @property decorators have only been made compatible since Python 3.9: [https://docs.python.org/3/howto/descriptor.html#class-methods] > The code path for {{hasattr(type(self.f), '__get__')}} was added in Python > 3.9 and makes it possible for > [{{classmethod()}}|https://docs.python.org/3/library/functions.html#classmethod] > to support chained decorators. Moreover, @classmethod + @property waq also affected by a bug which has been been fixed in Python 3.9.6: https://github.com/python/cpython/pull/28838 So we had to use a temporary workaround by defining a custom decorator named @classproperty to fix SPARK-38870. *What should be done* Once support for Python 3.8 and "<=3.9.5" is officially dropped, we should remove this @classproperty workaround and replace it with @classmethod + @property (cf TODOs in the [pyspark.sql.session module|https://github.com/apache/spark/pull/36161/files]). *Further details* _Python 3.8 will end-of-life is 2024/10._ was: The proper bugfix of SPARK-38870 uses a feature of Python 3.9 and had to use a workaround for earlier version (cf TODOs in the [pyspark.sql.session module|https://github.com/apache/spark/pull/36161/files]) Once support for Python 3.8 is officially dropped, we should remove this workaround. _Python 3.8 will end-of-life is 2024/10._ > Clean "classproperty" workaround in pyspark.sql.session once support for > Python 3.8 is dropped > ---------------------------------------------------------------------------------------------- > > Key: SPARK-38912 > URL: https://issues.apache.org/jira/browse/SPARK-38912 > Project: Spark > Issue Type: Improvement > Components: PySpark, SQL > Affects Versions: 3.2.1 > Reporter: Furcy Pin > Priority: Minor > > *Context* > The proper bugfix of SPARK-38870 would be to use the @classmethod + @property > decorators on the SparkSession.builder attribute to make it return a new > builder every time like in Scala. > However, @classmethod + @property decorators have only been made compatible > since Python 3.9: > [https://docs.python.org/3/howto/descriptor.html#class-methods] > > The code path for {{hasattr(type(self.f), '__get__')}} was added in Python > > 3.9 and makes it possible for > > [{{classmethod()}}|https://docs.python.org/3/library/functions.html#classmethod] > > to support chained decorators. > Moreover, @classmethod + @property waq also affected by a bug which has been > been fixed in Python 3.9.6: > https://github.com/python/cpython/pull/28838 > So we had to use a temporary workaround by defining a custom decorator named > @classproperty to fix SPARK-38870. > *What should be done* > Once support for Python 3.8 and "<=3.9.5" is officially dropped, we should > remove this @classproperty workaround and replace it with @classmethod + > @property (cf TODOs in the [pyspark.sql.session > module|https://github.com/apache/spark/pull/36161/files]). > > *Further details* > _Python 3.8 will end-of-life is 2024/10._ -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org