[
https://issues.apache.org/jira/browse/SPARK-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-7478:
---------------------------------
Description:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770]. Also to get around this problem I had to suggest
creating a singleton instance -
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
was:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
> Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
> ---------------------------------------------------------------------------
>
> Key: SPARK-7478
> URL: https://issues.apache.org/jira/browse/SPARK-7478
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Priority: Blocker
>
> Having a SQLContext singleton would make it easier for applications to use a
> lazily instantiated single shared instance of SQLContext when needed. It
> would avoid problems like
> 1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
> SQLContext}} multiple times created different contexts while overriding the
> reference to previous context, leading to issues like registered temp tables
> going missing.
> 2. In Streaming, creating SQLContext directly leads to
> serialization/deserialization issues when attempting to recover from DStream
> checkpoints. See [SPARK-6770]. Also to get around this problem I had to
> suggest creating a singleton instance -
> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala
> This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
> singleton instance of SQLContext using either a given SparkContext or a given
> SparkConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
