This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3f5645a235af docs(schema-evolution): show how to disable
hive.metastore.disallow.incompatible.col.type.changes (#19551)
3f5645a235af is described below
commit 3f5645a235af9a7857dc1adfd956f2223b7ddb98
Author: deepakpanda93 <[email protected]>
AuthorDate: Mon Aug 17 10:40:26 2026 +0530
docs(schema-evolution): show how to disable
hive.metastore.disallow.incompatible.col.type.changes (#19551)
* docs(schema-evolution): show how to disable
hive.metastore.disallow.incompatible.col.type.changes
The schema evolution page told users to disable
hive.metastore.disallow.incompatible.col.type.changes when an ALTER TABLE
hits "The following columns have types incompatible with the existing
columns in their respective positions", but never showed how. That matters
for adding a column with FIRST or AFTER, which the same page documents as
supported.
The metastore evaluates the check against its own conf
(HiveAlterHandler#alterTable reads handler.getConf()), so where the property
has to be set depends on the metastore the engine talks to. Replace the bare
note with a subsection covering all three mechanisms:
- Spark with an embedded metastore: the metastore shares the Spark JVM, so
the spark.hadoop. prefix reaches it.
- Remote HMS: set it in the server's hive-site.xml and restart, or
- override it per connection with the metaconf: prefix, which routes to
IMetaStoreClient#setMetaConf. The property is a metaConfVars entry, so the
server accepts the override.
Applied to next and to version-1.2.0, the current released docs.
Closes #15648.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(schema-evolution): note that the metaconf: form is Hive/Beeline only
Review feedback on apache/hudi#19551: the page is read mostly by Spark SQL
users, and nothing stopped them from trying the metaconf: override against a
remote metastore.
Verified on Spark 3.5.7 + Hive 3.1.3 that this fails in a particularly
misleading way. spark-sql accepts
set
metaconf:hive.metastore.disallow.incompatible.col.type.changes=false;
without error and echoes the value back as false when read, but stores it as
an ordinary SQLConf entry whose key is literally
"metaconf:hive.metastore..."
Nothing routes it to the metastore, and the subsequent ALTER still fails
with
"The following columns have types incompatible with the existing columns in
their respective positions". Only Hive's SetProcessor treats the prefix
specially, calling Hive#setMetaConf -> IMetaStoreClient#setMetaConf.
Add a note that the form is Hive/Beeline only and point Spark users on a
remote metastore at the hive-site.xml option instead.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(schema-evolution): scope the metaconf: override to Beeline
Follow-up to the earlier metaconf: note on apache/hudi#19551, which said the
form works "from Hive or Beeline". Testing against Hive 3.1.3 shows the Hive
CLI half of that is wrong.
Over HiveServer2 (Beeline) it works: `set metaconf:...=false` followed by an
incompatible ALTER succeeded, taking the table from (id, name, ts) to
(id, age, name, ts).
The legacy Hive CLI reaches the metastore -- reading the value back with
`set metaconf:<key>` returns false where it returned true before -- but the
ALTER is still rejected, because the connection running the DDL is not the
one
carrying the override.
Name HiveServer2 explicitly and fold the Hive CLI into the same advice
already
given for Spark SQL: use the hive-site.xml option instead.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
website/docs/schema_evolution.md | 44 ++++++++++++++++++++--
.../version-1.2.0/schema_evolution.md | 44 ++++++++++++++++++++--
2 files changed, 82 insertions(+), 6 deletions(-)
diff --git a/website/docs/schema_evolution.md b/website/docs/schema_evolution.md
index 91465606cad6..bd89c24a4d63 100755
--- a/website/docs/schema_evolution.md
+++ b/website/docs/schema_evolution.md
@@ -3,7 +3,7 @@ title: Schema Evolution
keywords: [hudi, incremental, batch, stream, processing, schema, evolution]
summary: In this page, we will discuss schema evolution support in Hudi.
toc: true
-last_modified_at: 2022-04-27T15:59:57-04:00
+last_modified_at: 2026-08-05T22:49:08+05:30
---
Schema evolution is an essential aspect of data management, and Hudi supports
schema evolution on write out-of-the-box,
@@ -168,9 +168,47 @@ ALTER TABLE tableName RENAME COLUMN old_columnName TO
new_columnName
ALTER TABLE table1 RENAME COLUMN a.b.c TO x
```
+### Disabling the Hive metastore column type compatibility check
+
+For tables registered in the Hive metastore, the metastore compares the new
column list against the existing one
+position by position and rejects the `ALTER TABLE` when the types in a given
position are not compatible. Schema changes
+that alter the type or the position of an existing column, such as adding a
column with `FIRST` or `AFTER`, can
+therefore fail with:
+
+`The following columns have types incompatible with the existing columns in
their respective positions`
+
+Setting `hive.metastore.disallow.incompatible.col.type.changes` to `false`
skips this check. The metastore evaluates the
+check, so where you set the property depends on the metastore the engine talks
to.
+
+When Spark runs its own embedded metastore (no `hive.metastore.uris`
configured), the metastore shares the Spark JVM, so
+pass the property with Spark's `spark.hadoop.` prefix when you start the job:
+
+```shell
+--conf
'spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false'
+```
+
+When the engine talks to a remote Hive metastore service, the property has to
take effect on the server. Either set it
+in the metastore's `hive-site.xml` and restart the service:
+
+```xml
+<property>
+ <name>hive.metastore.disallow.incompatible.col.type.changes</name>
+ <value>false</value>
+</property>
+```
+
+or override it for a single Beeline session with the `metaconf:` prefix, which
pushes the value to the metastore for
+that connection:
+
+```sql
+set metaconf:hive.metastore.disallow.incompatible.col.type.changes=false;
+```
+
:::note
-When using hive metastore, please disable
`hive.metastore.disallow.incompatible.col.type.changes` if you encounter this
error:
-`The following columns have types incompatible with the existing columns in
their respective positions`.
+Only Hive's SQL processor handles the `metaconf:` prefix, and the override
applies over HiveServer2. Spark SQL accepts
+the same statement and echoes the value back, but stores it as an ordinary
session property that never reaches the
+metastore. The legacy Hive CLI does send the value to the metastore, but not
on the connection that runs the
+`ALTER`. In both cases, use the `hive-site.xml` option above instead.
:::
## Schema Evolution in Action
diff --git a/website/versioned_docs/version-1.2.0/schema_evolution.md
b/website/versioned_docs/version-1.2.0/schema_evolution.md
index 91465606cad6..bd89c24a4d63 100755
--- a/website/versioned_docs/version-1.2.0/schema_evolution.md
+++ b/website/versioned_docs/version-1.2.0/schema_evolution.md
@@ -3,7 +3,7 @@ title: Schema Evolution
keywords: [hudi, incremental, batch, stream, processing, schema, evolution]
summary: In this page, we will discuss schema evolution support in Hudi.
toc: true
-last_modified_at: 2022-04-27T15:59:57-04:00
+last_modified_at: 2026-08-05T22:49:08+05:30
---
Schema evolution is an essential aspect of data management, and Hudi supports
schema evolution on write out-of-the-box,
@@ -168,9 +168,47 @@ ALTER TABLE tableName RENAME COLUMN old_columnName TO
new_columnName
ALTER TABLE table1 RENAME COLUMN a.b.c TO x
```
+### Disabling the Hive metastore column type compatibility check
+
+For tables registered in the Hive metastore, the metastore compares the new
column list against the existing one
+position by position and rejects the `ALTER TABLE` when the types in a given
position are not compatible. Schema changes
+that alter the type or the position of an existing column, such as adding a
column with `FIRST` or `AFTER`, can
+therefore fail with:
+
+`The following columns have types incompatible with the existing columns in
their respective positions`
+
+Setting `hive.metastore.disallow.incompatible.col.type.changes` to `false`
skips this check. The metastore evaluates the
+check, so where you set the property depends on the metastore the engine talks
to.
+
+When Spark runs its own embedded metastore (no `hive.metastore.uris`
configured), the metastore shares the Spark JVM, so
+pass the property with Spark's `spark.hadoop.` prefix when you start the job:
+
+```shell
+--conf
'spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false'
+```
+
+When the engine talks to a remote Hive metastore service, the property has to
take effect on the server. Either set it
+in the metastore's `hive-site.xml` and restart the service:
+
+```xml
+<property>
+ <name>hive.metastore.disallow.incompatible.col.type.changes</name>
+ <value>false</value>
+</property>
+```
+
+or override it for a single Beeline session with the `metaconf:` prefix, which
pushes the value to the metastore for
+that connection:
+
+```sql
+set metaconf:hive.metastore.disallow.incompatible.col.type.changes=false;
+```
+
:::note
-When using hive metastore, please disable
`hive.metastore.disallow.incompatible.col.type.changes` if you encounter this
error:
-`The following columns have types incompatible with the existing columns in
their respective positions`.
+Only Hive's SQL processor handles the `metaconf:` prefix, and the override
applies over HiveServer2. Spark SQL accepts
+the same statement and echoes the value back, but stores it as an ordinary
session property that never reaches the
+metastore. The legacy Hive CLI does send the value to the metastore, but not
on the connection that runs the
+`ALTER`. In both cases, use the `hive-site.xml` option above instead.
:::
## Schema Evolution in Action
