[
https://issues.apache.org/jira/browse/SPARK-15712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated SPARK-15712:
-------------------------------
Description:
For proper temp table support, I am proposing to create a temp dir for every
{{SparkSession}}. This can be done through the following changes:
# We add a root temp dir (the path is configurable).
# For every {{SparkSession}}, we create a temp dir for this {{SparkSession}} in
the root temp dir.
To make the semantic of {{CREATE TEMP TABLE ... USING ... OPTIONS ... AS
SELECT}} easier to understand, I am proposing the following changes:
# We only allow {{FileFormat}} data sources used by this command.
# We make {{USING}} clause optional and by default we use the data source set
in {{spark.sql.sources.default}}.
# We do not allow users to add path in the {{OPTIONS}}
# We save the temp table in the temp location associated with the current
{{SparkSession}}.
was:
For proper temp table support, I am proposing to create a temp dir for every
SparkSession. This can be done through the following changes:
1. We add a root temp dir (the path is configurable).
2. For every SparkSession, we create a temp dir for this SparkSession in the
root temp dir.
To make the semantic of CREATE TEMP TABLE ... USING ... OPTIONS ... AS SELECT
easier to understand, I am proposing the following changes:
1. We only allow HadoopFsRelations used by this command.
2. We make USING clause a optional clause and by default we use the format set
in spark.sql.sources.default.
3. We do not allow users to add path in the OPTIONS
4. We save the temp table in the temp location associated with the current
SparkSession.
> Proper temp table support
> -------------------------
>
> Key: SPARK-15712
> URL: https://issues.apache.org/jira/browse/SPARK-15712
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Yin Huai
>
> For proper temp table support, I am proposing to create a temp dir for every
> {{SparkSession}}. This can be done through the following changes:
> # We add a root temp dir (the path is configurable).
> # For every {{SparkSession}}, we create a temp dir for this {{SparkSession}}
> in the root temp dir.
> To make the semantic of {{CREATE TEMP TABLE ... USING ... OPTIONS ... AS
> SELECT}} easier to understand, I am proposing the following changes:
> # We only allow {{FileFormat}} data sources used by this command.
> # We make {{USING}} clause optional and by default we use the data source set
> in {{spark.sql.sources.default}}.
> # We do not allow users to add path in the {{OPTIONS}}
> # We save the temp table in the temp location associated with the current
> {{SparkSession}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
