hudi-agent commented on code in PR #19551: URL: https://github.com/apache/hudi/pull/19551#discussion_r3735254276
########## website/docs/schema_evolution.md: ########## @@ -168,10 +168,41 @@ ALTER TABLE tableName RENAME COLUMN old_columnName TO new_columnName ALTER TABLE table1 RENAME COLUMN a.b.c TO x ``` -:::note -When using hive metastore, please disable `hive.metastore.disallow.incompatible.col.type.changes` if you encounter this error: -`The following columns have types incompatible with the existing columns in their respective positions`. -::: +### Disabling the Hive metastore column type compatibility check + +For tables registered in the Hive metastore, the metastore compares the new column list against the existing one +position by position and rejects the `ALTER TABLE` when the types in a given position are not compatible. Schema changes +that alter the type or the position of an existing column, such as adding a column with `FIRST` or `AFTER`, can +therefore fail with: + +`The following columns have types incompatible with the existing columns in their respective positions` + +Setting `hive.metastore.disallow.incompatible.col.type.changes` to `false` skips this check. The metastore evaluates the +check, so where you set the property depends on the metastore the engine talks to. + +When Spark runs its own embedded metastore (no `hive.metastore.uris` configured), the metastore shares the Spark JVM, so +pass the property with Spark's `spark.hadoop.` prefix when you start the job: + +```shell +--conf 'spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false' +``` + +When the engine talks to a remote Hive metastore service, the property has to take effect on the server. Either set it +in the metastore's `hive-site.xml` and restart the service: + +```xml +<property> + <name>hive.metastore.disallow.incompatible.col.type.changes</name> + <value>false</value> +</property> +``` + +or override it for a single Hive or Beeline session with the `metaconf:` prefix, which pushes the value to the metastore Review Comment: 🤖 Since this page primarily documents schema evolution for Spark SQL users, it might help to make explicit that the `set metaconf:...` override only works from a Hive/Beeline session (routed to the HMS via `SetProcessor`), and is *not* available from `spark-sql` against a remote HMS. A Spark user hitting this error on a remote metastore would need the server-side `hive-site.xml` option above, since the client conf does not reach the metastore's handler conf. A one-line note clarifying that the metaconf trick is Hive/Beeline-only would prevent Spark users from trying it and being confused when it has no effect. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
