GitHub user RussellSpitzer opened a pull request: https://github.com/apache/spark/pull/21988
[SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark ## What changes were proposed in this pull request? Previously Pyspark used the private constructor for SparkSession when building that object. This resulted in a SparkSession without checking the sql.extensions parameter for additional session extensions. To fix this we instead use the Session.builder() path as SparkR uses, this loads the extensions and allows their use in PySpark. ## How was this patch tested? This was manually tested by passing a class to spark.sql.extensions and making sure it's included strategies appeared in the spark._jsparkSession.sessionState.planner.strategies list. We could add a automatic test but i'm not very familiar with the Pyspark Testing framework. But I would be glad to implement that if requested. You can merge this pull request into a Git repository by running: $ git pull https://github.com/RussellSpitzer/spark SPARK-25003-branch-2.2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21988.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21988 ---- commit 9c6e0671bf7311e2647e774c8d247c43037fc12c Author: Russell Spitzer <russell.spitzer@...> Date: 2018-08-03T15:48:15Z [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark Previously Pyspark used the private constructor for SparkSession when building that object. This resulted in a SparkSession without checking the sql.extensions parameter for additional session extensions. To fix this we instead use the Session.builder() path as SparkR uses, this loads the extensions and allows their use in PySpark. ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org