[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887296
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql
---
@@ -0,0 +1,29 @@
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added';
+
+DESC formatted table_with_comment;
+
+-- ALTER TABLE BY MODIFYING COMMENT
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet");
--- End diff --
Nit: `set tblproperties` -> `SET TBLPROPERTIES`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887356
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql
---
@@ -0,0 +1,29 @@
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added';
+
+DESC formatted table_with_comment;
+
+-- ALTER TABLE BY MODIFYING COMMENT
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet");
+
+DESC formatted table_with_comment;
+
+-- DROP TEST TABLE
+DROP TABLE table_with_comment;
+
+-- CREATE TABLE WITHOUT COMMENT
+CREATE TABLE table_comment (a STRING, b INT) USING parquet;
+
+DESC formatted table_comment;
--- End diff --
Nit: `formatted` -> `FORMATTED`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887266
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql
---
@@ -0,0 +1,29 @@
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added';
+
+DESC formatted table_with_comment;
+
+-- ALTER TABLE BY MODIFYING COMMENT
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet");
+
+DESC formatted table_with_comment;
+
+-- DROP TEST TABLE
+DROP TABLE table_with_comment;
+
+-- CREATE TABLE WITHOUT COMMENT
+CREATE TABLE table_comment (a STRING, b INT) USING parquet;
+
+DESC formatted table_comment;
+
+-- ALTER TABLE BY ADDING COMMENT
+ALTER TABLE table_comment set tblproperties(comment = "added comment");
--- End diff --
Nit: `set tblproperties` -> `SET TBLPROPERTIES`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887172
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql
---
@@ -0,0 +1,29 @@
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added';
+
+DESC formatted table_with_comment;
+
+-- ALTER TABLE BY MODIFYING COMMENT
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet");
+
+DESC formatted table_with_comment;
+
+-- DROP TEST TABLE
+DROP TABLE table_with_comment;
+
+-- CREATE TABLE WITHOUT COMMENT
+CREATE TABLE table_comment (a STRING, b INT) USING parquet;
+
+DESC formatted table_comment;
+
+-- ALTER TABLE BY ADDING COMMENT
+ALTER TABLE table_comment set tblproperties(comment = "added comment");
+
+DESC formatted table_comment;
+
+-- ALTER UNSET PROPERTIES COMMENT
+ALTER TABLE table_comment UNSET TBLPROPERTIES IF EXISTS ('comment');
+
+DESC formatted table_comment;
--- End diff --
Nit: `formatted` -> `FORMATTED`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887127
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql
---
@@ -0,0 +1,29 @@
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added';
+
+DESC formatted table_with_comment;
+
+-- ALTER TABLE BY MODIFYING COMMENT
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet");
+
+DESC formatted table_with_comment;
--- End diff --
Nit: `formatted` -> `FORMATTED`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887153
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql
---
@@ -0,0 +1,29 @@
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added';
+
+DESC formatted table_with_comment;
+
+-- ALTER TABLE BY MODIFYING COMMENT
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet");
+
+DESC formatted table_with_comment;
+
+-- DROP TEST TABLE
+DROP TABLE table_with_comment;
+
+-- CREATE TABLE WITHOUT COMMENT
+CREATE TABLE table_comment (a STRING, b INT) USING parquet;
+
+DESC formatted table_comment;
+
+-- ALTER TABLE BY ADDING COMMENT
+ALTER TABLE table_comment set tblproperties(comment = "added comment");
+
+DESC formatted table_comment;
--- End diff --
Nit: `formatted` -> `FORMATTED`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r114887110 --- Diff: sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql --- @@ -0,0 +1,29 @@ +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'added'; + +DESC formatted table_with_comment; --- End diff -- Nit: `formatted` -> `FORMATTED` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114887000
--- Diff:
sql/core/src/test/resources/sql-tests/results/describe-table-after-alter-table.sql.out
---
@@ -0,0 +1,161 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 12
+
+
+-- !query 0
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added'
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+DESC formatted table_with_comment
+-- !query 1 schema
+struct
+-- !query 1 output
+# col_name data_type comment
+a string
+b int
+c string
+d string
+
+# Detailed Table Information
+Database default
+Table table_with_comment
+Created [not included in comparison]
+Last Access [not included in comparison]
+Type MANAGED
+Provider parquet
+Commentadded
+Location [not included in
comparison]sql/core/spark-warehouse/table_with_comment
+
+
+-- !query 2
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet")
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+DESC formatted table_with_comment
--- End diff --
Nit: `formatted` -> `FORMATTED`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114886955
--- Diff:
sql/core/src/test/resources/sql-tests/results/describe-table-after-alter-table.sql.out
---
@@ -0,0 +1,161 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 12
+
+
+-- !query 0
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added'
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+DESC formatted table_with_comment
+-- !query 1 schema
+struct
+-- !query 1 output
+# col_name data_type comment
+a string
+b int
+c string
+d string
+
+# Detailed Table Information
+Database default
+Table table_with_comment
+Created [not included in comparison]
+Last Access [not included in comparison]
+Type MANAGED
+Provider parquet
+Commentadded
+Location [not included in
comparison]sql/core/spark-warehouse/table_with_comment
+
+
+-- !query 2
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet")
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+DESC formatted table_with_comment
+-- !query 3 schema
+struct
+-- !query 3 output
+# col_name data_type comment
+a string
+b int
+c string
+d string
+
+# Detailed Table Information
+Database default
+Table table_with_comment
+Created [not included in comparison]
+Last Access [not included in comparison]
+Type MANAGED
+Provider parquet
+Commentmodified comment
+Properties [type=parquet]
+Location [not included in
comparison]sql/core/spark-warehouse/table_with_comment
+
+
+-- !query 4
+DROP TABLE table_with_comment
+-- !query 4 schema
+struct<>
+-- !query 4 output
+
+
+
+-- !query 5
+CREATE TABLE table_comment (a STRING, b INT) USING parquet
+-- !query 5 schema
+struct<>
+-- !query 5 output
+
+
+
+-- !query 6
+DESC formatted table_comment
--- End diff --
Nit: `formatted` -> `FORMATTED`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114886889
--- Diff:
sql/core/src/test/resources/sql-tests/results/describe-table-after-alter-table.sql.out
---
@@ -0,0 +1,161 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 12
+
+
+-- !query 0
+CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING)
USING parquet COMMENT 'added'
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+DESC formatted table_with_comment
+-- !query 1 schema
+struct
+-- !query 1 output
+# col_name data_type comment
+a string
+b int
+c string
+d string
+
+# Detailed Table Information
+Database default
+Table table_with_comment
+Created [not included in comparison]
+Last Access [not included in comparison]
+Type MANAGED
+Provider parquet
+Commentadded
+Location [not included in
comparison]sql/core/spark-warehouse/table_with_comment
+
+
+-- !query 2
+ALTER TABLE table_with_comment set tblproperties("comment"= "modified
comment", "type"= "parquet")
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+DESC formatted table_with_comment
+-- !query 3 schema
+struct
+-- !query 3 output
+# col_name data_type comment
+a string
+b int
+c string
+d string
+
+# Detailed Table Information
+Database default
+Table table_with_comment
+Created [not included in comparison]
+Last Access [not included in comparison]
+Type MANAGED
+Provider parquet
+Commentmodified comment
+Properties [type=parquet]
+Location [not included in
comparison]sql/core/spark-warehouse/table_with_comment
+
+
+-- !query 4
+DROP TABLE table_with_comment
+-- !query 4 schema
+struct<>
+-- !query 4 output
+
+
+
+-- !query 5
+CREATE TABLE table_comment (a STRING, b INT) USING parquet
+-- !query 5 schema
+struct<>
+-- !query 5 output
+
+
+
+-- !query 6
+DESC formatted table_comment
+-- !query 6 schema
+struct
+-- !query 6 output
+# col_name data_type comment
+a string
+b int
+
+# Detailed Table Information
+Database default
+Table table_comment
+Created [not included in comparison]
+Last Access [not included in comparison]
+Type MANAGED
+Provider parquet
+Location [not included in comparison]sql/core/spark-warehouse/table_comment
+
+
+-- !query 7
+ALTER TABLE table_comment set tblproperties(comment = "added comment")
--- End diff --
Nit: `set tblproperties` -> `SET TBLPROPERTIES`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, p
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114886756
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -222,7 +222,6 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
} else {
tableDefinition.storage.locationUri
}
-
--- End diff --
Please revert it back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r114886656 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala --- @@ -267,8 +271,11 @@ case class AlterTableUnsetPropertiesCommand( } } } +// if 'comment' key is present in the seq of keys which needs to be unset then reset the table +// level comment with none. --- End diff -- Nit: How about? > If `comment` is in the table property, we reset it to None --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r114885074
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -216,8 +216,9 @@ class InMemoryCatalog(
} else {
tableDefinition
}
-
- catalog(db).tables.put(table, new TableDesc(tableWithLocation))
+ catalog(db).tables.put(table, new TableDesc(tableWithLocation
+ .copy(properties = tableWithLocation
+ .properties.filter(c1 => c1._1 != "comment"
--- End diff --
Nit:
```
val tableProp = tableWithLocation.properties.filter(c1 => c1._1 !=
"comment")
catalog(db).tables.put(table, new
TableDesc(tableWithLocation.copy(properties = tableProp)))
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r113616846 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -295,7 +295,9 @@ class InMemoryCatalog( assert(tableDefinition.identifier.database.isDefined) val db = tableDefinition.identifier.database.get requireTableExists(db, tableDefinition.identifier.table) -catalog(db).tables(tableDefinition.identifier.table).table = tableDefinition +val updatedProperties = tableDefinition.properties.filter(kv => kv._1 != "comment") +val newTableDefinition = tableDefinition.copy(properties = updatedProperties) +catalog(db).tables(tableDefinition.identifier.table).table = newTableDefinition --- End diff -- Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r113599057 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -295,7 +295,9 @@ class InMemoryCatalog( assert(tableDefinition.identifier.database.isDefined) val db = tableDefinition.identifier.database.get requireTableExists(db, tableDefinition.identifier.table) -catalog(db).tables(tableDefinition.identifier.table).table = tableDefinition +val updatedProperties = tableDefinition.properties.filter(kv => kv._1 != "comment") +val newTableDefinition = tableDefinition.copy(properties = updatedProperties) +catalog(db).tables(tableDefinition.identifier.table).table = newTableDefinition --- End diff -- sure, i will check on this point and will update the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r113500327 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -295,7 +295,9 @@ class InMemoryCatalog( assert(tableDefinition.identifier.database.isDefined) val db = tableDefinition.identifier.database.get requireTableExists(db, tableDefinition.identifier.table) -catalog(db).tables(tableDefinition.identifier.table).table = tableDefinition +val updatedProperties = tableDefinition.properties.filter(kv => kv._1 != "comment") +val newTableDefinition = tableDefinition.copy(properties = updatedProperties) +catalog(db).tables(tableDefinition.identifier.table).table = newTableDefinition --- End diff -- We should not duplicate `comment` in both CatalogTable field and table properties. Please correct it if you found it in the code. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r113427428 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -295,7 +295,9 @@ class InMemoryCatalog( assert(tableDefinition.identifier.database.isDefined) val db = tableDefinition.identifier.database.get requireTableExists(db, tableDefinition.identifier.table) -catalog(db).tables(tableDefinition.identifier.table).table = tableDefinition +val updatedProperties = tableDefinition.properties.filter(kv => kv._1 != "comment") +val newTableDefinition = tableDefinition.copy(properties = updatedProperties) +catalog(db).tables(tableDefinition.identifier.table).table = newTableDefinition --- End diff -- @gatorsmile @wzhfy as i noticed for hive tables already the 'comment' property is been filtered while getting the hive table metadata(reading) in **HiveClientImpl.scala** -- > **getTableOption()** API. so i think it will be better if we can do the same for **InMemoryCatalogs** type table, while reading the table metadata we can exclude the 'comment', i think if we exclude from table definition while alter command then there can be an inconsistency, eg: In create table we will persist the comment in metadata and while altering we are not. Please correct me if i am going in wrong direction, Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r113373050 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -295,7 +295,9 @@ class InMemoryCatalog( assert(tableDefinition.identifier.database.isDefined) val db = tableDefinition.identifier.database.get requireTableExists(db, tableDefinition.identifier.table) -catalog(db).tables(tableDefinition.identifier.table).table = tableDefinition +val updatedProperties = tableDefinition.properties.filter(kv => kv._1 != "comment") +val newTableDefinition = tableDefinition.copy(properties = updatedProperties) +catalog(db).tables(tableDefinition.identifier.table).table = newTableDefinition --- End diff -- This only fixes the issue in InMemoryCatalog. We still have a hole in HiveExternalCatalog. Could you move the fixes to `AlterTableSetPropertiesCommand `? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112840921 --- Diff: sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql --- @@ -0,0 +1,29 @@ +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment'; + +DESC formatted table_with_comment; + +-- ALTER TABLE BY MODIFYING COMMENT +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment"); --- End diff -- modified the testcase as per comment, added one more proeprty while altering the table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user sujith71955 commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112838259 --- Diff: sql/core/src/test/resources/sql-tests/results/describe-table-after-alter-table.sql.out --- @@ -0,0 +1,162 @@ +-- Automatically generated by SQLQueryTestSuite +-- Number of queries: 12 + + +-- !query 0 +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment' +-- !query 0 schema +struct<> +-- !query 0 output + + + +-- !query 1 +DESC formatted table_with_comment +-- !query 1 schema +struct +-- !query 1 output +# col_name data_type comment +a string +b int +c string +d string + +# Detailed Table Information +Database default +Table table_with_comment +Created [not included in comparison] +Last Access [not included in comparison] +Type MANAGED +Provider parquet +Commenttable_comment +Location [not included in comparison]sql/core/spark-warehouse/table_with_comment + + +-- !query 2 +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment") +-- !query 2 schema +struct<> +-- !query 2 output + + + +-- !query 3 +DESC formatted table_with_comment +-- !query 3 schema +struct +-- !query 3 output +# col_name data_type comment +a string +b int +c string +d string + +# Detailed Table Information +Database default +Table table_with_comment +Created [not included in comparison] +Last Access [not included in comparison] +Type MANAGED +Provider parquet +Commentmodified comment +Properties [comment=modified comment] --- End diff -- yes, i saw for hive tables already its been taken care in **HiveClientImpl.scala** class where "comment" is getting filtered from table properties, also for parquet table same has to be taken care. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112824647 --- Diff: sql/core/src/test/resources/sql-tests/results/describe-table-after-alter-table.sql.out --- @@ -0,0 +1,162 @@ +-- Automatically generated by SQLQueryTestSuite +-- Number of queries: 12 + + +-- !query 0 +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment' +-- !query 0 schema +struct<> +-- !query 0 output + + + +-- !query 1 +DESC formatted table_with_comment +-- !query 1 schema +struct +-- !query 1 output +# col_name data_type comment +a string +b int +c string +d string + +# Detailed Table Information +Database default +Table table_with_comment +Created [not included in comparison] +Last Access [not included in comparison] +Type MANAGED +Provider parquet +Commenttable_comment +Location [not included in comparison]sql/core/spark-warehouse/table_with_comment + + +-- !query 2 +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment") +-- !query 2 schema +struct<> +-- !query 2 output + + + +-- !query 3 +DESC formatted table_with_comment +-- !query 3 schema +struct +-- !query 3 output +# col_name data_type comment +a string +b int +c string +d string + +# Detailed Table Information +Database default +Table table_with_comment +Created [not included in comparison] +Last Access [not included in comparison] +Type MANAGED +Provider parquet +Commentmodified comment +Properties [comment=modified comment] --- End diff -- We should remove `comment` from `Properties ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17649#discussion_r112824524
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -267,8 +271,15 @@ case class AlterTableUnsetPropertiesCommand(
}
}
}
+// if 'comment' key is present in the seq of keys which needs to be
unset then reset the table
+// level comment with none.
+val tableComment = if (propKeys.contains("comment")) {
+ None
+} else {
+ table.properties.get("comment")
+}
--- End diff --
Nit:
```Scala
val comment = if (propKeys.contains("comment")) None else
table.properties.get("comment")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112635039 --- Diff: sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql --- @@ -0,0 +1,29 @@ +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment'; + +DESC formatted table_with_comment; + +-- ALTER TABLE BY MODIFYING COMMENT +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment"); --- End diff -- nvm I missed your previous discussion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112634259 --- Diff: sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql --- @@ -0,0 +1,29 @@ +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment'; + +DESC formatted table_with_comment; + +-- ALTER TABLE BY MODIFYING COMMENT +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment"); --- End diff -- rename `describe-table-after-alter-table.sql` to `alter-table-comment`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112632107 --- Diff: sql/core/src/test/resources/sql-tests/inputs/describe-table-after-alter-table.sql --- @@ -0,0 +1,29 @@ +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment'; + +DESC formatted table_with_comment; + +-- ALTER TABLE BY MODIFYING COMMENT +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment"); --- End diff -- I think it would be better to also have other properties with comment in this test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17649: [SPARK-20380][SQL] Output table comment for DESC ...
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/17649#discussion_r112630905 --- Diff: sql/core/src/test/resources/sql-tests/results/describe-table-after-alter-table.sql.out --- @@ -0,0 +1,162 @@ +-- Automatically generated by SQLQueryTestSuite +-- Number of queries: 12 + + +-- !query 0 +CREATE TABLE table_with_comment (a STRING, b INT, c STRING, d STRING) USING parquet COMMENT 'table_comment' +-- !query 0 schema +struct<> +-- !query 0 output + + + +-- !query 1 +DESC formatted table_with_comment +-- !query 1 schema +struct +-- !query 1 output +# col_name data_type comment +a string +b int +c string +d string + +# Detailed Table Information +Database default +Table table_with_comment +Created [not included in comparison] +Last Access [not included in comparison] +Type MANAGED +Provider parquet +Commenttable_comment +Location [not included in comparison]sql/core/spark-warehouse/table_with_comment + + +-- !query 2 +ALTER TABLE table_with_comment set tblproperties(comment = "modified comment") +-- !query 2 schema +struct<> +-- !query 2 output + + + +-- !query 3 +DESC formatted table_with_comment +-- !query 3 schema +struct +-- !query 3 output +# col_name data_type comment +a string +b int +c string +d string + +# Detailed Table Information +Database default +Table table_with_comment +Created [not included in comparison] +Last Access [not included in comparison] +Type MANAGED +Provider parquet +Commentmodified comment +Properties [comment=modified comment] --- End diff -- Comment string i.e. "modified comment" is in both `Properties` and `Comment`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
