[
https://issues.apache.org/jira/browse/HIVE-30133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30133:
----------------------------------
Summary: Columnar shuffle: a batch with repeating partition columns travels
as one block (was: Vectorized ReduceSink hashes a repeating partition key for
every row)
> Columnar shuffle: a batch with repeating partition columns travels as one
> block
> -------------------------------------------------------------------------------
>
> Key: HIVE-30133
> URL: https://issues.apache.org/jira/browse/HIVE-30133
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
>
> VectorReduceSinkObjectHashOperator extracts and hashes the partition columns
> for every row. When every partition column repeats across the batch, every
> row gets the same hashCode, so it is now computed once per batch. The bucket
> number and _bucket_number are still computed per row.
> Iceberg V3 merge-on-read DELETE distributes by FILE__PATH, which repeats
> within a batch. In a microbenchmark of the operator: 107.7 -> 53.4 ns/row for
> a repeating 150-byte string key, -24% for a repeating bigint, unchanged when
> the key does not repeat.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
