weiliang hao created SPARK-41241: ------------------------------------ Summary: Use Hive and Spark SQL to modify table field comment, the modified results of Hive cannot be queried using Spark SQL Key: SPARK-41241 URL: https://issues.apache.org/jira/browse/SPARK-41241 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.0, 3.2.0, 3.1.0, 3.0.0 Reporter: weiliang hao
---HIVE--- > create table table_test(id int); > alter table table_test change column id id int comment "hive comment"; > desc formatted table_test; {code:java} +-------------------------------+----------------------------------------------------+----------------------------------------------------+ | col_name | data_type | comment | +-------------------------------+----------------------------------------------------+----------------------------------------------------+ | # col_name | data_type | comment | | id | int | hive comment | | | NULL | NULL | | # Detailed Table Information | NULL | NULL | | Database: | default | NULL | | OwnerType: | USER | NULL | | Owner: | anonymous | NULL | | CreateTime: | Wed Nov 23 23:06:41 CST 2022 | NULL | | LastAccessTime: | UNKNOWN | NULL | | Retention: | 0 | NULL | | Location: | hdfs://localhost:8020/warehouse/tablespace/managed/hive/table_test | NULL | | Table Type: | MANAGED_TABLE | NULL | | Table Parameters: | NULL | NULL | | | COLUMN_STATS_ACCURATE | {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}} | | | bucketing_version | 2 | | | last_modified_by | anonymous | | | last_modified_time | 1669216665 | | | numFiles | 0 | | | numRows | 0 | | | rawDataSize | 0 | | | totalSize | 0 | | | transactional | true | | | transactional_properties | default | | | transient_lastDdlTime | 1669216665 | | | NULL | NULL | | # Storage Information | NULL | NULL | | SerDe Library: | org.apache.hadoop.hive.ql.io.orc.OrcSerde | NULL | | InputFormat: | org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | NULL | | OutputFormat: | org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL | | Compressed: | No | NULL | | Num Buckets: | -1 | NULL | | Bucket Columns: | [] | NULL | | Sort Columns: | [] | NULL | | Storage Desc Params: | NULL | NULL | | | serialization.format | 1 | +-------------------------------+----------------------------------------------------+----------------------------------------------------+ {code} ---SPARK--- > alter table table_test change column id id int comment "spark comment"; > desc formatted table_test; {code:java} +-------------------------------+----------------------------------------------------+--------------+ | col_name | data_type | comment | +-------------------------------+----------------------------------------------------+--------------+ | id | int | spark comment | | | | | | # Detailed Table Information | | | | Catalog | spark_catalog | | | Database | default | | | Table | table_test | | | Owner | anonymous | | | Created Time | Wed Nov 23 23:06:41 CST 2022 | | | Last Access | UNKNOWN | | | Created By | Spark 2.2 or prior | | | Type | MANAGED | | | Provider | hive | | | Table Properties | [bucketing_version=2, last_modified_by=anonymous, last_modified_time=1669216665, transactional=true, transactional_properties=default, transient_lastDdlTime=1669216711] | | | Location | hdfs://localhost:8020/warehouse/tablespace/managed/hive/table_test | | | Serde Library | org.apache.hadoop.hive.ql.io.orc.OrcSerde | | | InputFormat | org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | | | OutputFormat | org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | | | Storage Properties | [serialization.format=1] | | | Partition Provider | Catalog | | {code} ---HIVE--- > alter table table_test change column id id int comment "hive new comment"; ---SPARK--- > desc formatted table_test; {code:java} +-------------------------------+----------------------------------------------------+--------------+ | col_name | data_type | comment | +-------------------------------+----------------------------------------------------+--------------+ | id | int | spark comment | | | | | | # Detailed Table Information | | | | Catalog | spark_catalog | | | Database | default | | | Table | table_test | | | Owner | anonymous | | | Created Time | Wed Nov 23 23:06:41 CST 2022 | | | Last Access | UNKNOWN | | | Created By | Spark 2.2 or prior | | | Type | MANAGED | | | Provider | hive | | | Table Properties | [bucketing_version=2, last_modified_by=anonymous, last_modified_time=1669216736, transactional=true, transactional_properties=default, transient_lastDdlTime=1669216736] | | | Location | hdfs://localhost:8020/warehouse/tablespace/managed/hive/table_test | | | Serde Library | org.apache.hadoop.hive.ql.io.orc.OrcSerde | | | InputFormat | org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | | | OutputFormat | org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | | | Storage Properties | [serialization.format=1] | | | Partition Provider | Catalog | | +-------------------------------+----------------------------------------------------+--------------+ {code} Alternately modify table field comments with hive and spark,the modified results of hive cannot be queried using spark sql -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org