[GitHub] [spark] gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"
gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table" URL: https://github.com/apache/spark/pull/24938#discussion_r37675 ## File path: docs/sql-migration-guide.md ## @@ -328,6 +328,8 @@ license: | - Since Spark 3.0, `SHOW TBLPROPERTIES` will cause `AnalysisException` if the table does not exist. In Spark version 2.4 and earlier, this scenario caused `NoSuchTableException`. Also, `SHOW TBLPROPERTIES` on a temporary view will cause `AnalysisException`. In Spark version 2.4 and earlier, it returned an empty result. + - Since Spark 3.0, `SHOW CREATE TABLE` will always return Spark DDL, even when the given table is a Hive serde table. For Hive DDL, please use `SHOW CREATE TABLE AS SERDE` command instead. Review comment: `For Hive DDL` -> `For generating Hive DDL` ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"
gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL]
Hive DDL to Spark DDL conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#discussion_r376756495
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##
@@ -1017,16 +1067,154 @@ case class ShowCreateTableCommand(table:
TableIdentifier) extends RunnableComman
// TODO: [SPARK-28692] unify this after we unify the
// CREATE TABLE syntax for hive serde and data source table.
- val stmt = if (DDLUtils.isDatasourceTable(tableMetadata)) {
-showCreateDataSourceTable(tableMetadata)
+ val metadata = if (DDLUtils.isDatasourceTable(tableMetadata)) {
+tableMetadata
} else {
-showCreateHiveTable(tableMetadata)
+// For a Hive serde table, we try to convert it to Spark DDL.
+if (tableMetadata.unsupportedFeatures.nonEmpty) {
+ throw new AnalysisException(
+"Failed to execute SHOW CREATE TABLE against table " +
+ s"${tableMetadata.identifier}, which is created by Hive and uses
the " +
+ "following unsupported feature(s)\n" +
+ tableMetadata.unsupportedFeatures.map(" - " + _).mkString("\n")
+ )
+}
+
+if (tableMetadata.tableType == VIEW) {
+ throw new AnalysisException("Hive view isn't supported by SHOW
CREATE TABLE")
+}
+
+if
("true".equalsIgnoreCase(tableMetadata.properties.getOrElse("transactional",
"false"))) {
+ throw new AnalysisException(
+"SHOW CREATE TABLE doesn't support transactional Hive table")
Review comment:
The same here. Let end users know what are the workaround, i.e., new syntax.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"
gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL]
Hive DDL to Spark DDL conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#discussion_r376756401
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##
@@ -1017,16 +1067,154 @@ case class ShowCreateTableCommand(table:
TableIdentifier) extends RunnableComman
// TODO: [SPARK-28692] unify this after we unify the
// CREATE TABLE syntax for hive serde and data source table.
- val stmt = if (DDLUtils.isDatasourceTable(tableMetadata)) {
-showCreateDataSourceTable(tableMetadata)
+ val metadata = if (DDLUtils.isDatasourceTable(tableMetadata)) {
+tableMetadata
} else {
-showCreateHiveTable(tableMetadata)
+// For a Hive serde table, we try to convert it to Spark DDL.
+if (tableMetadata.unsupportedFeatures.nonEmpty) {
+ throw new AnalysisException(
+"Failed to execute SHOW CREATE TABLE against table " +
+ s"${tableMetadata.identifier}, which is created by Hive and uses
the " +
+ "following unsupported feature(s)\n" +
Review comment:
Can we improve the exception message? Let end users know what are the new
syntax for CREATE HIVE SERDE table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"
gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL]
Hive DDL to Spark DDL conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#discussion_r376756408
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
##
@@ -1017,16 +1067,154 @@ case class ShowCreateTableCommand(table:
TableIdentifier) extends RunnableComman
// TODO: [SPARK-28692] unify this after we unify the
// CREATE TABLE syntax for hive serde and data source table.
- val stmt = if (DDLUtils.isDatasourceTable(tableMetadata)) {
-showCreateDataSourceTable(tableMetadata)
+ val metadata = if (DDLUtils.isDatasourceTable(tableMetadata)) {
+tableMetadata
} else {
-showCreateHiveTable(tableMetadata)
+// For a Hive serde table, we try to convert it to Spark DDL.
+if (tableMetadata.unsupportedFeatures.nonEmpty) {
+ throw new AnalysisException(
+"Failed to execute SHOW CREATE TABLE against table " +
+ s"${tableMetadata.identifier}, which is created by Hive and uses
the " +
+ "following unsupported feature(s)\n" +
+ tableMetadata.unsupportedFeatures.map(" - " + _).mkString("\n")
+ )
+}
+
+if (tableMetadata.tableType == VIEW) {
+ throw new AnalysisException("Hive view isn't supported by SHOW
CREATE TABLE")
Review comment:
Can we just create Spark View?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"
gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table" URL: https://github.com/apache/spark/pull/24938#discussion_r372542927 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -196,7 +196,7 @@ statement | SHOW PARTITIONS multipartIdentifier partitionSpec? #showPartitions | SHOW identifier? FUNCTIONS (LIKE? (multipartIdentifier | pattern=STRING))? #showFunctions -| SHOW CREATE TABLE multipartIdentifier #showCreateTable +| SHOW CREATE TABLE multipartIdentifier (AS SPARK)? #showCreateTable Review comment: Yes! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"
gatorsmile commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table" URL: https://github.com/apache/spark/pull/24938#discussion_r363444039 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -196,7 +196,7 @@ statement | SHOW PARTITIONS multipartIdentifier partitionSpec? #showPartitions | SHOW identifier? FUNCTIONS (LIKE? (multipartIdentifier | pattern=STRING))? #showFunctions -| SHOW CREATE TABLE multipartIdentifier #showCreateTable +| SHOW CREATE TABLE multipartIdentifier (AS SPARK)? #showCreateTable Review comment: After rethinking it, let us make it more aggressive here. Instead of creating Spark native tables for the existing Hive serde tables, we can try to always show how to create Spark native tables if possible. This will further simplify the migration from Hive to Spark. To the existing Spark users who prefer to keeping Hive serde formats, we can introduce a new option `AS SERDE` which will keep the behaviors in Spark 2.4 or prior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
