[
https://issues.apache.org/jira/browse/SPARK-41241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640305#comment-17640305
]
Erik Krogen commented on SPARK-41241:
-------------------------------------
I think, as a general rule, it's not safe to modify Spark datasource tables
using Hive. Spark stores a bunch of extra (Spark-specific) metadata in the
properties of the table, and Hive will not modify this metadata since it is not
aware of it. So I would consider this behavior to be expected?
> Use Hive and Spark SQL to modify table field comment, the modified results of
> Hive cannot be queried using Spark SQL
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-41241
> URL: https://issues.apache.org/jira/browse/SPARK-41241
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0
> Reporter: weiliang hao
> Priority: Major
>
> ---HIVE---
> > create table table_test(id int);
> > alter table table_test change column id id int comment "hive comment";
> > desc formatted table_test;
> {code:java}
> +-------------------------------+----------------------------------------------------+----------------------------------------------------+
> | col_name | data_type
> | comment |
> +-------------------------------+----------------------------------------------------+----------------------------------------------------+
> | # col_name | data_type
> | comment |
> | id | int
> | hive comment |
> | | NULL
> | NULL |
> | # Detailed Table Information | NULL
> | NULL |
> | Database: | default
> | NULL |
> | OwnerType: | USER
> | NULL |
> | Owner: | anonymous
> | NULL |
> | CreateTime: | Wed Nov 23 23:06:41 CST 2022
> | NULL |
> | LastAccessTime: | UNKNOWN
> | NULL |
> | Retention: | 0
> | NULL |
> | Location: |
> hdfs://localhost:8020/warehouse/tablespace/managed/hive/table_test | NULL
> |
> | Table Type: | MANAGED_TABLE
> | NULL |
> | Table Parameters: | NULL
> | NULL |
> | | COLUMN_STATS_ACCURATE
> | {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\"}} |
> | | bucketing_version
> | 2 |
> | | last_modified_by
> | anonymous |
> | | last_modified_time
> | 1669216665 |
> | | numFiles
> | 0 |
> | | numRows
> | 0 |
> | | rawDataSize
> | 0 |
> | | totalSize
> | 0 |
> | | transactional
> | true |
> | | transactional_properties
> | default |
> | | transient_lastDdlTime
> | 1669216665 |
> | | NULL
> | NULL |
> | # Storage Information | NULL
> | NULL |
> | SerDe Library: | org.apache.hadoop.hive.ql.io.orc.OrcSerde
> | NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | NULL
> |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL
> |
> | Compressed: | No
> | NULL |
> | Num Buckets: | -1
> | NULL |
> | Bucket Columns: | []
> | NULL |
> | Sort Columns: | []
> | NULL |
> | Storage Desc Params: | NULL
> | NULL |
> | | serialization.format
> | 1 |
> +-------------------------------+----------------------------------------------------+----------------------------------------------------+
> {code}
> ---SPARK---
> > alter table table_test change column id id int comment "spark comment";
> > desc formatted table_test;
> {code:java}
> +-------------------------------+----------------------------------------------------+--------------+
> | col_name | data_type
> | comment |
> +-------------------------------+----------------------------------------------------+--------------+
> | id | int
> | spark comment |
> | |
> | |
> | # Detailed Table Information |
> | |
> | Catalog | spark_catalog
> | |
> | Database | default
> | |
> | Table | table_test
> | |
> | Owner | anonymous
> | |
> | Created Time | Wed Nov 23 23:06:41 CST 2022
> | |
> | Last Access | UNKNOWN
> | |
> | Created By | Spark 2.2 or prior
> | |
> | Type | MANAGED
> | |
> | Provider | hive
> | |
> | Table Properties | [bucketing_version=2,
> last_modified_by=anonymous, last_modified_time=1669216665,
> transactional=true, transactional_properties=default,
> transient_lastDdlTime=1669216711] | |
> | Location |
> hdfs://localhost:8020/warehouse/tablespace/managed/hive/table_test |
> |
> | Serde Library | org.apache.hadoop.hive.ql.io.orc.OrcSerde
> | |
> | InputFormat |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | |
> | OutputFormat |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | |
> | Storage Properties | [serialization.format=1]
> | |
> | Partition Provider | Catalog
> | | {code}
> ---HIVE---
> > alter table table_test change column id id int comment "hive new comment";
>
> ---SPARK---
> > desc formatted table_test;
> {code:java}
> +-------------------------------+----------------------------------------------------+--------------+
> | col_name | data_type
> | comment |
> +-------------------------------+----------------------------------------------------+--------------+
> | id | int
> | spark comment |
> | |
> | |
> | # Detailed Table Information |
> | |
> | Catalog | spark_catalog
> | |
> | Database | default
> | |
> | Table | table_test
> | |
> | Owner | anonymous
> | |
> | Created Time | Wed Nov 23 23:06:41 CST 2022
> | |
> | Last Access | UNKNOWN
> | |
> | Created By | Spark 2.2 or prior
> | |
> | Type | MANAGED
> | |
> | Provider | hive
> | |
> | Table Properties | [bucketing_version=2,
> last_modified_by=anonymous, last_modified_time=1669216736,
> transactional=true, transactional_properties=default,
> transient_lastDdlTime=1669216736] | |
> | Location |
> hdfs://localhost:8020/warehouse/tablespace/managed/hive/table_test |
> |
> | Serde Library | org.apache.hadoop.hive.ql.io.orc.OrcSerde
> | |
> | InputFormat |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | |
> | OutputFormat |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | |
> | Storage Properties | [serialization.format=1]
> | |
> | Partition Provider | Catalog
> | |
> +-------------------------------+----------------------------------------------------+--------------+
> {code}
>
> Alternately modify table field comments with hive and spark，the modified
> results of hive cannot be queried using spark sql
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
