Github user clockfly commented on the issue:
https://github.com/apache/spark/pull/14750
Here is my two cents:
### 1. About case sensitivity
Before this change, the column name we stored in Hive meta store is always
lower case for Hive serde table (Hive Metastore converts the column to lower
case). After this change, now, the CatalogTable can also contains upper case
columns. It changes the user behavior. For example, command
CreateTableLikeCommand will copy the schema from old table. If the old schema
contains upper case columns, it means the new table now also contains upper
case columns.
### 2. It may create data consistency problem when storing table schema,
partition columns, to table properties. We'd better have single source of
truth for table schema data.
For example, AlterTableAddPartitionCommand triggers call
`HiveClientImpl.createPartitions`, which basically changes the partition
columns. Current impl of `HiveClientImpl.createPartitions` is calling into hive
directly.
```
override def createPartitions(
db: String,
table: String,
parts: Seq[CatalogTablePartition],
ignoreIfExists: Boolean): Unit = withHiveState {
shim.createPartitions(client, db, table, parts, ignoreIfExists)
}
```
If we put the partition columns in table properties, then we use call
`ALTER TABLE table ADD [IF NOT EXISTS] PARTITION spec [LOCATION 'loc1']`, the
table properties are not updated, which means we will not be able to track this
partition column change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
