[
https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832718#comment-16832718
]
Sujith Chacko edited comment on SPARK-27617 at 5/3/19 6:20 PM:
---------------------------------------------------------------
HIVE does not "externalise" table if LOCATION specified wihtout "EXTERNAL'
keyword, Impala neither Why does Apache Spark introduce different behavior
here?
I feel we can be compatible with hive/impala in this case, as i mentioned or
else we block certain use-cases as i mentioned above.
was (Author: s71955):
HIVE does not "externalise" table if LOCATION specified wihtout "EXTERNAL'
keyword, Impala neither Why does Apache Spark introduce different behaviour?
> Not able to specify LOCATION for internal table
> -----------------------------------------------
>
> Key: SPARK-27617
> URL: https://issues.apache.org/jira/browse/SPARK-27617
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.0, 2.2.0, 2.3.0, 2.4.0, 3.0.0
> Reporter: Sujith Chacko
> Priority: Major
>
> In Spark whenever user specifies location uri in create table without
> external keyword the table is treated as external table .
> Because of this behavior following following problems has been observed
> a) user will not able to set an external location for a managed table.
> b) compatibility issue with hive/impala where the system allow managed table
> to specify location uri if user created table without 'EXTERNAL' keyword.
> {code:java}
> scala> spark.sql("""CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION
> 'D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13'""");
> -chgrp: 'HTIPL-23270\None' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> res15: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("desc formatted ext2").show(false)
>
> +-----------------------------+---------------------------------------------------------------++-------
> |col_name|data_type|comment|
> +-----------------------------+---------------------------------------------------------------++-------
> |name|string|null|
> | | | |
> | # Detailed Table Information| | |
> |Database|default| |
> |Table|ext2| |
> |Owner|Administrator| |
> |Created Time|Wed May 01 21:52:57 IST 2019| |
> |Last Access|Thu Jan 01 05:30:00 IST 1970| |
> |Created By|Spark 2.4.1| |
> |Type|EXTERNAL| |
> |Provider|hive| |
> |Table Properties|[transient_lastDdlTime=1556727777]| |
> |Location|file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13| |
> |Serde Library|org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe| |
> |InputFormat|org.apache.hadoop.mapred.TextInputFormat| |
> |OutputFormat|org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat| |
> |Storage Properties|[serialization.format=1]| |
> |Partition Provider|Catalog| |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
