[
https://issues.apache.org/jira/browse/HIVE-30184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30184:
----------------------------------
Description:
VariantParquetFilters.buildNullPredicate always builds
FilterApi.eq/notEq(binaryColumn(path), null), so the
row-group filter declares the shredded typed_value leaf as BINARY. For a
non-string leaf Parquet rejects it:
{code}
FilterPredicate column: payload.typed_value.event_id.typed_value's declared
type (org.apache.parquet.io.api.Binary)
does not match the schema found in file metadata. Column
payload.typed_value.event_id.typed_value is of type: INT32
{code}
Repro: an Iceberg v3 table with variant.shredding.enabled=true and integer
values in $.event_id;
SELECT id FROM t WHERE variant_get(payload, '$.event_id', 'int') IS NULL fails
in both execution modes.
Fix: build the null predicate with the leaf's physical type, or skip row-group
pruning for IS NULL on non-binary
leaves.
> Iceberg: variant_get(..., 'int'|'long'|'double'|'boolean') IS [NOT] NULL
> fails on shredded tables
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-30184
> URL: https://issues.apache.org/jira/browse/HIVE-30184
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> VariantParquetFilters.buildNullPredicate always builds
> FilterApi.eq/notEq(binaryColumn(path), null), so the
> row-group filter declares the shredded typed_value leaf as BINARY. For a
> non-string leaf Parquet rejects it:
> {code}
> FilterPredicate column: payload.typed_value.event_id.typed_value's declared
> type (org.apache.parquet.io.api.Binary)
> does not match the schema found in file metadata. Column
> payload.typed_value.event_id.typed_value is of type: INT32
> {code}
> Repro: an Iceberg v3 table with variant.shredding.enabled=true and integer
> values in $.event_id;
> SELECT id FROM t WHERE variant_get(payload, '$.event_id', 'int') IS NULL
> fails in both execution modes.
> Fix: build the null predicate with the leaf's physical type, or skip
> row-group pruning for IS NULL on non-binary
> leaves.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
