[
https://issues.apache.org/jira/browse/HIVE-30166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30166:
----------------------------------
Description:
Since HIVE-29770, {{select v}} returns a variant as JSON, but the column type
reported for it is still Hive's internal struct:
{code:sql}
create table t (v variant) stored by iceberg tblproperties
('format-version'='3');
insert into t values (parse_json('{"name":"John","age":30}'));
select v from t; -- {"age":30,"name":"John"}
select typeof(v) from t; -- struct<metadata:binary,value:binary>
{code}
For {{select v}}, JDBC {{ResultSetMetaData}} returns {{getColumnTypeName}} =
struct and {{getColumnType}} = {{Types.STRUCT}}. This does not depend on
{{hive.serdes.using.metastore.for.schema}}: the result is the same with
{{HiveIcebergSerDe}} in the list.
*Cause:* {{VariantObjectInspector extends StandardStructObjectInspector}}, so
it inherits {{Category.STRUCT}} and the struct type name. The planner types
table columns from the SerDe ObjectInspector, so {{variant}} ->
{{VariantTypeInfo}} -> {{VariantObjectInspector}} -> {{StructTypeInfo}}.
{{VariantTypeInfo}} and {{Category.VARIANT}} exist (HIVE-29184), but the
inspector
never reports them.
*Compatibility:* once the result schema says VARIANT, HS2 sends
{{TTypeId.VARIANT_TYPE}} (24, added by HIVE-29770) in GetResultSetMetadata.
Every released driver fails to decode an unknown type id ({{TProtocolException:
Required field 'type' is unset!}}). So HS2 must report VARIANT only to sessions
that negotiated {{HIVE_CLI_SERVICE_PROTOCOL_V12}} and keep reporting struct to
older ones. Today the highest server-side gate is V9 (serialize-in-tasks).
Changing the inspector's category or type name affects every consumer that
handles it as a struct. That includes the HIVE-29770 JSON decode in
{{SerDeUtils}}, which recognises a variant by the struct type name.
> Iceberg: report VARIANT instead of struct<metadata:binary,value:binary> as
> the result type of variant columns
> --------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30166
> URL: https://issues.apache.org/jira/browse/HIVE-30166
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> Since HIVE-29770, {{select v}} returns a variant as JSON, but the column type
> reported for it is still Hive's internal struct:
>
> {code:sql}
>
>
> create table t (v variant) stored by iceberg tblproperties
> ('format-version'='3');
>
> insert into t values (parse_json('{"name":"John","age":30}'));
>
>
> select v from t; -- {"age":30,"name":"John"}
>
>
> select typeof(v) from t; -- struct<metadata:binary,value:binary>
>
>
> {code}
>
>
> For {{select v}}, JDBC {{ResultSetMetaData}} returns {{getColumnTypeName}}
> = struct and {{getColumnType}} = {{Types.STRUCT}}. This does not depend on
> {{hive.serdes.using.metastore.for.schema}}: the result is the same with
> {{HiveIcebergSerDe}} in the list.
>
> *Cause:* {{VariantObjectInspector extends StandardStructObjectInspector}}, so
> it inherits {{Category.STRUCT}} and the struct type name. The planner types
> table columns from the SerDe ObjectInspector, so {{variant}} ->
> {{VariantTypeInfo}} -> {{VariantObjectInspector}} -> {{StructTypeInfo}}.
> {{VariantTypeInfo}} and {{Category.VARIANT}} exist (HIVE-29184), but the
> inspector
> never reports them.
>
>
>
>
>
> *Compatibility:* once the result schema says VARIANT, HS2 sends
> {{TTypeId.VARIANT_TYPE}} (24, added by HIVE-29770) in GetResultSetMetadata.
> Every released driver fails to decode an unknown type id
> ({{TProtocolException: Required field 'type' is unset!}}). So HS2 must report
> VARIANT only to sessions that negotiated {{HIVE_CLI_SERVICE_PROTOCOL_V12}}
> and keep reporting struct to older ones. Today the highest server-side gate
> is V9 (serialize-in-tasks).
>
>
>
>
> Changing the inspector's category or type name affects every consumer that
> handles it as a struct. That includes the HIVE-29770 JSON decode in
> {{SerDeUtils}}, which recognises a variant by the struct type name.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
