[
https://issues.apache.org/jira/browse/SPARK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15444981#comment-15444981
]
Jeff Zhang commented on SPARK-17261:
------------------------------------
It works if you change 'sc._instantiatedContext = None' to
'SparkSession._instantiatedContext = None'. The problem here is that
SQLContext/HIveContext is created from SparkSession, although you create
HiveContext, SparkSession will still be created first underneath. And here the
SparkContext is stopped, but SparkSession don't know that, so it still use the
stopped SparkContext.
> Using HiveContext after re-creating SparkContext in Spark 2.0 throws
> "Java.lang.illegalStateException: Cannot call methods on a stopped
> sparkContext"
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17261
> URL: https://issues.apache.org/jira/browse/SPARK-17261
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.0.0
> Environment: Amazon AWS EMR 5.0
> Reporter: Rahul Jain
> Fix For: 2.0.0
>
>
> After stopping SparkSession if we recreate it and use HiveContext in it. it
> will throw error.
> Steps to reproduce:
> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> spark.sql("show databases")
> spark.stop()
> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> spark.sql("show databases")
> "Java.lang.illegalStateException: Cannot call methods on a stopped
> sparkContext"
> Above error occurs only in case of Pyspark not in SparkShell
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
