deepakpanda93 commented on code in PR #19551: URL: https://github.com/apache/hudi/pull/19551#discussion_r3735328392
########## website/docs/schema_evolution.md: ########## @@ -168,10 +168,41 @@ ALTER TABLE tableName RENAME COLUMN old_columnName TO new_columnName ALTER TABLE table1 RENAME COLUMN a.b.c TO x ``` -:::note -When using hive metastore, please disable `hive.metastore.disallow.incompatible.col.type.changes` if you encounter this error: -`The following columns have types incompatible with the existing columns in their respective positions`. -::: +### Disabling the Hive metastore column type compatibility check + +For tables registered in the Hive metastore, the metastore compares the new column list against the existing one +position by position and rejects the `ALTER TABLE` when the types in a given position are not compatible. Schema changes +that alter the type or the position of an existing column, such as adding a column with `FIRST` or `AFTER`, can +therefore fail with: + +`The following columns have types incompatible with the existing columns in their respective positions` + +Setting `hive.metastore.disallow.incompatible.col.type.changes` to `false` skips this check. The metastore evaluates the +check, so where you set the property depends on the metastore the engine talks to. + +When Spark runs its own embedded metastore (no `hive.metastore.uris` configured), the metastore shares the Spark JVM, so +pass the property with Spark's `spark.hadoop.` prefix when you start the job: + +```shell +--conf 'spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false' +``` + +When the engine talks to a remote Hive metastore service, the property has to take effect on the server. Either set it +in the metastore's `hive-site.xml` and restart the service: + +```xml +<property> + <name>hive.metastore.disallow.incompatible.col.type.changes</name> + <value>false</value> +</property> +``` + +or override it for a single Hive or Beeline session with the `metaconf:` prefix, which pushes the value to the metastore Review Comment: Good catch — addressed in 5aabd164fb13. I verified this on the Spark 3.5.7 + Hive 3.1.3 (standalone HMS) stack I used for the rest of the PR, and it fails more misleadingly than described. `spark-sql` **accepts** the statement without error and **echoes the value back** when you read it: ``` spark-sql> set metaconf:hive.metastore.disallow.incompatible.col.type.changes=false; spark-sql> set metaconf:hive.metastore.disallow.incompatible.col.type.changes; metaconf:hive.metastore.disallow.incompatible.col.type.changes false ``` …yet the subsequent `ALTER TABLE ... ADD COLUMNS (age int AFTER id)` still fails with `The following columns have types incompatible with the existing columns in their respective positions : age`. Spark just stores it as an ordinary `SQLConf` entry whose key is literally `metaconf:hive.metastore...`; only Hive's `SetProcessor` treats the prefix specially and calls `Hive#setMetaConf` → `IMetaStoreClient#setMetaConf`. So it reads as if it worked, which is the worst case for a user. Added a note under that snippet saying the form is Hive/Beeline only and pointing Spark users on a remote metastore at the `hive-site.xml` option instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
