[jira] [Updated] (SPARK-7478) Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
[
https://issues.apache.org/jira/browse/SPARK-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-7478:
-
Description:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
was:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}}
Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
---
Key: SPARK-7478
URL: https://issues.apache.org/jira/browse/SPARK-7478
Project: Spark
Issue Type: New Feature
Components: SQL
Reporter: Tathagata Das
Assignee: Tathagata Das
Priority: Blocker
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It
would avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7478) Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
[ https://issues.apache.org/jira/browse/SPARK-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-7478: - Description: Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like 1. In REPL/notebook environment, rerunning the line val sqlContext = new SQLContext multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing. 2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See [SPARK-6770]. was: Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like 1. In REPL/notebook environment, rerunning the line val sqlContext = new SQLContext multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing. 2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See [SPARK-6770](https://issues.apache.org/jira/browse/SPARK-6770) Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext --- Key: SPARK-7478 URL: https://issues.apache.org/jira/browse/SPARK-7478 Project: Spark Issue Type: New Feature Components: SQL Reporter: Tathagata Das Assignee: Tathagata Das Priority: Blocker Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like 1. In REPL/notebook environment, rerunning the line val sqlContext = new SQLContext multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing. 2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See [SPARK-6770]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7478) Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
[ https://issues.apache.org/jira/browse/SPARK-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-7478: - Description: Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like 1. In REPL/notebook environment, rerunning the line val sqlContext = new SQLContext multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing. 2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See [SPARK-6770](https://issues.apache.org/jira/browse/SPARK-6770) was: Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like 1. In REPL/notebook environment, rerunning the line val sqlContext = new SQLContext multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing. 2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext --- Key: SPARK-7478 URL: https://issues.apache.org/jira/browse/SPARK-7478 Project: Spark Issue Type: New Feature Components: SQL Reporter: Tathagata Das Assignee: Tathagata Das Priority: Blocker Having a SQLContext singleton would make it easier for applications to use a lazily instantiated single shared instance of SQLContext when needed. It would avoid problems like 1. In REPL/notebook environment, rerunning the line val sqlContext = new SQLContext multiple times created different contexts while overriding the reference to previous context, leading to issues like registered temp tables going missing. 2. In Streaming, creating SQLContext directly leads to serialization/deserialization issues when attempting to recover from DStream checkpoints. See [SPARK-6770](https://issues.apache.org/jira/browse/SPARK-6770) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7478) Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
[
https://issues.apache.org/jira/browse/SPARK-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-7478:
-
Description:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}}
was:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {val sqlContext = new
SQLContext} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {SQLContext.getOrCreate}
Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
---
Key: SPARK-7478
URL: https://issues.apache.org/jira/browse/SPARK-7478
Project: Spark
Issue Type: New Feature
Components: SQL
Reporter: Tathagata Das
Assignee: Tathagata Das
Priority: Blocker
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It
would avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7478) Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
[
https://issues.apache.org/jira/browse/SPARK-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das updated SPARK-7478:
-
Description:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770]. Also to get around this problem I had to suggest
creating a singleton instance -
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
was:
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It would
avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770].
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
Add a SQLContext.getOrCreate to maintain a singleton instance of SQLContext
---
Key: SPARK-7478
URL: https://issues.apache.org/jira/browse/SPARK-7478
Project: Spark
Issue Type: New Feature
Components: SQL
Reporter: Tathagata Das
Assignee: Tathagata Das
Priority: Blocker
Having a SQLContext singleton would make it easier for applications to use a
lazily instantiated single shared instance of SQLContext when needed. It
would avoid problems like
1. In REPL/notebook environment, rerunning the line {{val sqlContext = new
SQLContext}} multiple times created different contexts while overriding the
reference to previous context, leading to issues like registered temp tables
going missing.
2. In Streaming, creating SQLContext directly leads to
serialization/deserialization issues when attempting to recover from DStream
checkpoints. See [SPARK-6770]. Also to get around this problem I had to
suggest creating a singleton instance -
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala
This can be solved by {{SQLContext.getOrCreate}} which get or creates a new
singleton instance of SQLContext using either a given SparkContext or a given
SparkConf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
