[
https://issues.apache.org/jira/browse/HIVE-30164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30164:
----------------------------------
Description:
HMS stores an Iceberg VARIANT column as {{variant}}, and so does the Iceberg
schema. HS2, however, derive Iceberg column types from the SerDe's
ObjectInspector, and {{VariantObjectInspector}} is a struct inspector.
{code:sql}
create table t (v variant) stored by iceberg tblproperties
('format-version'='3');
describe t; -- v
struct<metadata:binary,value:binary>
show create table t; -- `v` struct<`metadata`:binary,
`value`:binary>
show tblproperties t('current-schema'); -- ... "type":"variant" ...
{code}
JDBC {{DatabaseMetaData.getColumns}} with an embedded HMS also returns
TYPE_NAME {{struct<metadata:binary,value:binary>}} and DATA_TYPE
{{Types.STRUCT}}. A standalone HMS returns {{variant}}.
Reusing the SHOW CREATE TABLE column list creates an Iceberg {{struct<metadata:
binary, value: binary>}} column, so the variant type is silently lost.
*Cause:* HIVE-27694 added {{HiveIcebergSerDe}} to the MetastoreConf default of
{{metastore.serdes.using.metastore.for.schema}}, but not to HiveConf's
{hive.serdes.using.metastore.for.schema}}. {{MetastoreConf.getVar}} falls back
to the {{hive.*}} key, which HiveConf always sets. As a result, HS2 and an
embedded HMS read Iceberg columns from the deserializer, while a standalone HMS
reads the stored schema. {{HBaseSerDe}} (HIVE-29484) has the same drift; it was
not tested here.
*Second gap:* with {{HiveIcebergSerDe}} added to the HiveConf list, DESCRIBE
shows {{variant}}, but SHOW CREATE TABLE fails:
{noformat}
java.lang.RuntimeException: Unknown type: VARIANT
at
org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.formatType(DDLPlanUtils.java:972)
at
org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.getColumns(DDLPlanUtils.java:917)
at
org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.getCreateTableCommand(DDLPlanUtils.java:892)
at
org.apache.hadoop.hive.ql.ddl.table.create.show.ShowCreateTableOperation.execute(ShowCreateTableOperation.java:61)
{noformat}
*Proposed fix:* align the HiveConf default with MetastoreConf, and handle
VARIANT in {{DDLPlanUtils.formatType}}.
Query result types and {{typeof()}} still report struct after this fix; they
are tracked in HIVE-30166.
was:
HMS stores an Iceberg VARIANT column as {{variant}}, and so does the Iceberg
schema. HS2, however, derive Iceberg column types from the SerDe's
ObjectInspector, and {{VariantObjectInspector}} is a struct inspector.
{code:sql}
create table t (v variant) stored by iceberg tblproperties
('format-version'='3');
describe t; -- v
struct<metadata:binary,value:binary>
show create table t; -- `v` struct<`metadata`:binary,
`value`:binary>
show tblproperties t('current-schema'); -- ... "type":"variant" ...
{code}
JDBC {{DatabaseMetaData.getColumns}} with an embedded HMS also returns
TYPE_NAME {{struct<metadata:binary,value:binary>}} and DATA_TYPE
{{Types.STRUCT}}. A standalone HMS returns {{variant}}.
Reusing the SHOW CREATE TABLE column list creates an Iceberg {{struct<metadata:
binary, value: binary>}} column, so the variant type is silently lost.
*Cause:* HIVE-27694 added {{HiveIcebergSerDe}} to the MetastoreConf default of
{{metastore.serdes.using.metastore.for.schema}}, but not to HiveConf's
{hive.serdes.using.metastore.for.schema}}. {{MetastoreConf.getVar}} falls back
to the {{hive.*}} key, which HiveConf always sets. As a result, HS2 and an
embedded HMS read Iceberg columns from the deserializer, while a standalone HMS
reads the stored schema. {{HBaseSerDe}} (HIVE-29484) has the same drift; it was
not tested here.
*Second gap:* with {{HiveIcebergSerDe}} added to the HiveConf list, DESCRIBE
shows {{variant}}, but SHOW CREATE TABLE fails:
{noformat}
java.lang.RuntimeException: Unknown type: VARIANT
at
org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.formatType(DDLPlanUtils.java:972)
at
org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.getColumns(DDLPlanUtils.java:917)
at
org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.getCreateTableCommand(DDLPlanUtils.java:892)
at
org.apache.hadoop.hive.ql.ddl.table.create.show.ShowCreateTableOperation.execute(ShowCreateTableOperation.java:61)
{noformat}
*Proposed fix:* align the HiveConf default with MetastoreConf, and handle
VARIANT in {{DDLPlanUtils.formatType}}.
Query result types and {{typeof()}} still report struct after this fix; they
are tracked in <ticket 3>.
> Iceberg: DESCRIBE, SHOW CREATE TABLE and getColumns report VARIANT columns as
> struct<metadata:binary,value:binary>
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30164
> URL: https://issues.apache.org/jira/browse/HIVE-30164
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HMS stores an Iceberg VARIANT column as {{variant}}, and so does the Iceberg
> schema. HS2, however, derive Iceberg column types from the SerDe's
> ObjectInspector, and {{VariantObjectInspector}} is a struct inspector.
>
>
>
>
>
> {code:sql}
>
>
> create table t (v variant) stored by iceberg tblproperties
> ('format-version'='3');
>
> describe t; -- v
> struct<metadata:binary,value:binary>
>
> show create table t; -- `v` struct<`metadata`:binary,
> `value`:binary>
>
> show tblproperties t('current-schema'); -- ... "type":"variant" ...
>
>
> {code}
>
>
>
> JDBC {{DatabaseMetaData.getColumns}} with an embedded HMS also returns
> TYPE_NAME {{struct<metadata:binary,value:binary>}} and DATA_TYPE
> {{Types.STRUCT}}. A standalone HMS returns {{variant}}.
> Reusing the SHOW CREATE TABLE column list creates an Iceberg
> {{struct<metadata: binary, value: binary>}} column, so the variant type is
> silently lost.
>
>
>
> *Cause:* HIVE-27694 added {{HiveIcebergSerDe}} to the MetastoreConf default
> of {{metastore.serdes.using.metastore.for.schema}}, but not to HiveConf's
> {hive.serdes.using.metastore.for.schema}}. {{MetastoreConf.getVar}} falls
> back to the {{hive.*}} key, which HiveConf always sets. As a result, HS2 and
> an embedded HMS read Iceberg columns from the deserializer, while a
> standalone HMS reads the stored schema. {{HBaseSerDe}} (HIVE-29484) has the
> same drift; it was not tested here.
> *Second gap:* with {{HiveIcebergSerDe}} added to the HiveConf list, DESCRIBE
> shows {{variant}}, but SHOW CREATE TABLE fails:
>
> {noformat}
>
>
> java.lang.RuntimeException: Unknown type: VARIANT
>
>
> at
> org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.formatType(DDLPlanUtils.java:972)
>
>
> at
> org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.getColumns(DDLPlanUtils.java:917)
>
>
> at
> org.apache.hadoop.hive.ql.ddl.DDLPlanUtils.getCreateTableCommand(DDLPlanUtils.java:892)
>
>
> at
> org.apache.hadoop.hive.ql.ddl.table.create.show.ShowCreateTableOperation.execute(ShowCreateTableOperation.java:61)
>
> {noformat}
>
>
>
>
>
> *Proposed fix:* align the HiveConf default with MetastoreConf, and handle
> VARIANT in {{DDLPlanUtils.formatType}}.
>
> Query result types and {{typeof()}} still report struct after this fix;
> they are tracked in HIVE-30166.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
