Hi community,

We are using HiveCatalog (Iceberg 1.0 + Spark 3.2 + HMS 1.2.2 with MYSQL as
its DB) and we encountered a column type mismatch failure when trying to
update the table.

The root cause is when creating the Iceberg table with a column that has a
super long column type string, mysql seems to cap the type string and only
retains first 4000 characters. This results in the subsequent update to the
table with updated column type string mismatched (since the updated column
type string sent to HMS is uncapped).

I proposed below change https://github.com/apache/iceberg/pull/7222/files
"[Draft][HiveCatalog] Skip updating column schema when field schema string
is larger than maxHiveTablePropertySize". In this PR,  I am reusing the old
column schema that is stored in HMS instead of the updated schema.

I want to check if you are fine with this or if you have a better idea on
how to resolve this issue. Another approach is not fixing it on the Iceberg
level, rather enforcing HMS to fail loudly during the table creation.

Best,
Pucheng

Reply via email to