[
https://issues.apache.org/jira/browse/HIVE-30164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30164:
----------------------------------
Description:
HMS stores an Iceberg VARIANT column as {{variant}}, and so does the Iceberg
schema. HS2, however, derive Iceberg column types from the SerDe's
ObjectInspector, and {{VariantObjectInspector}} is a struct inspector.
{code:sql}
create table t (v variant) stored by iceberg tblproperties
('format-version'='3');
describe t; -- v
struct<metadata:binary,value:binary>
show create table t; -- `v` struct<`metadata`:binary,
`value`:binary>
show tblproperties t('current-schema'); -- ... "type":"variant" ...
{code}
JDBC {{DatabaseMetaData.getColumns}} with an embedded HMS also returns
TYPE_NAME {{struct<metadata:binary,value:binary>}} and DATA_TYPE
{{Types.STRUCT}}. A standalone HMS returns {{variant}}.
Reusing the SHOW CREATE TABLE column list creates an Iceberg {{struct<metadata:
binary, value: binary>}} column, so the variant type is silently lost.
*Cause:* HIVE-27694 added {{HiveIcebergSerDe}} to the MetastoreConf default of
{{metastore.serdes.using.metastore.for.schema}}, but not to HiveConf's
{hive.serdes.using.metastore.for.schema}}. {{MetastoreConf.getVar}} falls back
to the {{hive.*}} key, which HiveConf always sets. As a result, HS2 and an
embedded HMS read Iceberg columns from the deserializer, while a standalone HMS
reads the stored schema. {{HBaseSerDe}} (HIVE-29484) has the same drift; it was
not tested here.
was:
Ticket 2: standalone, Bug, HiveServer2
HS2: TINYINT columns fail with ClassCastException when
hive.server2.thrift.resultset.serialize.in.tasks=true
{code:sql}
create table t (b tinyint);
insert into t values (7);
Jump to bottom (click) ↓
> Iceberg: DESCRIBE, SHOW CREATE TABLE and getColumns report VARIANT columns as
> struct<metadata:binary,value:binary>
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30164
> URL: https://issues.apache.org/jira/browse/HIVE-30164
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HMS stores an Iceberg VARIANT column as {{variant}}, and so does the Iceberg
> schema. HS2, however, derive Iceberg column types from the SerDe's
> ObjectInspector, and {{VariantObjectInspector}} is a struct inspector.
>
>
>
>
>
> {code:sql}
>
>
> create table t (v variant) stored by iceberg tblproperties
> ('format-version'='3');
>
> describe t; -- v
> struct<metadata:binary,value:binary>
>
> show create table t; -- `v` struct<`metadata`:binary,
> `value`:binary>
>
> show tblproperties t('current-schema'); -- ... "type":"variant" ...
>
>
> {code}
>
>
>
> JDBC {{DatabaseMetaData.getColumns}} with an embedded HMS also returns
> TYPE_NAME {{struct<metadata:binary,value:binary>}} and DATA_TYPE
> {{Types.STRUCT}}. A standalone HMS returns {{variant}}.
> Reusing the SHOW CREATE TABLE column list creates an Iceberg
> {{struct<metadata: binary, value: binary>}} column, so the variant type is
> silently lost.
>
>
>
> *Cause:* HIVE-27694 added {{HiveIcebergSerDe}} to the MetastoreConf default
> of {{metastore.serdes.using.metastore.for.schema}}, but not to HiveConf's
> {hive.serdes.using.metastore.for.schema}}. {{MetastoreConf.getVar}} falls
> back to the {{hive.*}} key, which HiveConf always sets. As a result, HS2 and
> an embedded HMS read Iceberg columns from the deserializer, while a
> standalone HMS reads the stored schema. {{HBaseSerDe}} (HIVE-29484) has the
> same drift; it was not tested here.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
