[ 
https://issues.apache.org/jira/browse/SPARK-30856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-30856.
----------------------------------
    Fix Version/s: 3.1.0
       Resolution: Fixed

Issue resolved by pull request 27610
[https://github.com/apache/spark/pull/27610]

> SQLContext retains reference to unusable instance after SparkContext restarted
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-30856
>                 URL: https://issues.apache.org/jira/browse/SPARK-30856
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 2.4.5
>            Reporter: Alex Favaro
>            Priority: Major
>             Fix For: 3.1.0
>
>
> When the underlying SQLContext is instantiated for a SparkSession, the 
> instance is saved as a class attribute and returned from subsequent calls to 
> SQLContext.getOrCreate(). If the SparkContext is stopped and a new one 
> started, the SQLContext class attribute is never cleared so any code which 
> calls SQLContext.getOrCreate() will get a SQLContext with a reference to the 
> old, unusable SparkContext.
> A similar issue was identified and fixed for SparkSession in SPARK-19055, but 
> the fix did not change SQLContext as well. I ran into this because mllib 
> still 
> [uses|https://github.com/apache/spark/blob/master/python/pyspark/mllib/common.py#L105]
>  SQLContext.getOrCreate() under the hood.
> I've already written a fix for this, which I'll be sharing in a PR, that 
> clears the class attribute on SQLContext when the SparkSession is stopped. 
> Another option would be to deprecate SQLContext.getOrCreate() entirely since 
> the corresponding Scala 
> [method|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SQLContext.html#getOrCreate-org.apache.spark.SparkContext-]
>  is itself deprecated. That seems like a larger change for a relatively minor 
> issue, however.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to