[
https://issues.apache.org/jira/browse/HIVE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771853#comment-16771853
]
George Pachitariu commented on HIVE-20523:
------------------------------------------
Hi [~asinkovits], thank you for having a look on my patch :).
Can you please give me a concrete example when my solution will give less
consistent results, compared to your solution?
My code gets called for each row, so it sees everything.
The same is reading from the footer in parquet files (you see all the rows).
I initially went with this approach because I wanted to be consistent with the
Orc implementation (so that later we can merge the code together, instead of
always implementing things the Parquet way and duplicating the effort to do it
the Orc way).
Cheers.
> Improve table statistics for Parquet format
> -------------------------------------------
>
> Key: HIVE-20523
> URL: https://issues.apache.org/jira/browse/HIVE-20523
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
> Reporter: George Pachitariu
> Assignee: George Pachitariu
> Priority: Minor
> Attachments: HIVE-20523.1.patch, HIVE-20523.10.patch,
> HIVE-20523.11.patch, HIVE-20523.12.patch, HIVE-20523.2.patch,
> HIVE-20523.3.patch, HIVE-20523.4.patch, HIVE-20523.5.patch,
> HIVE-20523.6.patch, HIVE-20523.7.patch, HIVE-20523.8.patch,
> HIVE-20523.9.patch, HIVE-20523.patch
>
>
> Right now, in the table basic statistics, the *raw data size* for a row with
> any data type in the Parquet format is 1. This is an underestimated value
> when columns are complex data structures, like arrays.
> Having tables with underestimated raw data size makes Hive assign less
> containers (mappers/reducers) to it, making the overall query slower.
> Heavy underestimation also makes Hive choose MapJoin instead of the
> ShuffleJoin that can fail with OOM errors.
> In this patch, I compute the columns data size better, taking into account
> complex structures. I followed the Writer implementation for the ORC format.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
