[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211897#comment-16211897
]
Jacky Shen commented on SPARK-21785:
any comment [~dongjoon]?
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193209#comment-16193209
]
Jacky Shen commented on SPARK-21785:
Yes, we can create table from existing parquet file and it is a good solution.
but for the request itself, the user want to create table with the schema from
the given parquet file and specify a different location to the table. Maybe
more other table properties.
that's the "create table ... using parquet" cannot achieve.
{code:sql}
spark-sql> create table table2 using parquet options(path
'/Users/jacshen/project/git-rep/spark/sql/core/src/test/resources/test-data/dec-in-fixed-len.parquet')
location '/tmp/table1';
Error in query:
LOCATION and 'path' in OPTIONS are both used to indicate the custom table path,
you can only specify one of them.(line 1, pos 0)
{code}
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192433#comment-16192433
]
Takeshi Yamamuro commented on SPARK-21785:
--
{code}
scala> sql("""CREATE TABLE customer USING parquet OPTIONS (path
'/tmp/spark-tpcds/customer')""")
res0: org.apache.spark.sql.DataFrame = []
scala> spark.table("customer").printSchema
root
|-- c_customer_sk: integer (nullable = true)
|-- c_customer_id: string (nullable = true)
|-- c_current_cdemo_sk: integer (nullable = true)
|-- c_current_hdemo_sk: integer (nullable = true)
|-- c_current_addr_sk: integer (nullable = true)
|-- c_first_shipto_date_sk: integer (nullable = true)
|-- c_first_sales_date_sk: integer (nullable = true)
|-- c_salutation: string (nullable = true)
|-- c_first_name: string (nullable = true)
|-- c_last_name: string (nullable = true)
|-- c_preferred_cust_flag: string (nullable = true)
|-- c_birth_day: integer (nullable = true)
|-- c_birth_month: integer (nullable = true)
|-- c_birth_year: integer (nullable = true)
|-- c_birth_country: string (nullable = true)
|-- c_login: string (nullable = true)
|-- c_email_address: string (nullable = true)
|-- c_last_review_date: string (nullable = true)
{code}
Am I missing something?
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192430#comment-16192430
]
Jacky Shen commented on SPARK-21785:
from my understanding, "CREATE TABLE xxx USING parquet" won't get schema from
the file.
what [~liupengcheng] wants to get the schema from the given parquet file.
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192362#comment-16192362
]
Takeshi Yamamuro commented on SPARK-21785:
--
`CREATE TABLE xxx USING parquet OPTIONS (path 'xxx')` is not enough for your
case?
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192244#comment-16192244
]
Apache Spark commented on SPARK-21785:
--
User 'CrazyJacky' has created a pull request for this issue:
https://github.com/apache/spark/pull/19434
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137743#comment-16137743
]
liupengcheng commented on SPARK-21785:
--
[~dongjoon] I am not so familiar with this part of SparkSQL for now, so hope
someone else could do it! Many Thanks!
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137727#comment-16137727
]
Dongjoon Hyun commented on SPARK-21785:
---
Thanks. Good to know that. +1 for the idea. Will you make a PR?
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137726#comment-16137726
]
liupengcheng commented on SPARK-21785:
--
[~dongjoon] Impala now support this kind of syntax. So I think it's much more
convenient if SparkSQL can support it.
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21785) Support create table from a file schema
[
https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137316#comment-16137316
]
Dongjoon Hyun commented on SPARK-21785:
---
It looks interesting. Is there any other SQL engine supporting this?
> Support create table from a file schema
> ---
>
> Key: SPARK-21785
> URL: https://issues.apache.org/jira/browse/SPARK-21785
> Project: Spark
> Issue Type: Wish
> Components: SQL
>Affects Versions: 2.1.1, 2.2.0
>Reporter: liupengcheng
> Labels: sql
>
> Current spark doest not support creating a table from a file schema
> for example:
> {code:java}
> CREATE EXTERNAL TABLE IF NOT EXISTS test LIKE PARQUET
> '/user/test/abc/a.snappy.parquet' STORED AS PARQUET LOCATION
> '/user/test/def/';
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
