[
https://issues.apache.org/jira/browse/HIVE-29534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18071993#comment-18071993
]
Stamatis Zampetakis commented on HIVE-29534:
--------------------------------------------
As mentioned under the PR, the CBO optimization phase is affected heavily by
NDV statistics and as demonstrated by the tests there are quite a few join
order changes in the TPC-DS query plans. This change may cause noticeable
changes in TPC-DS benchmarks.
More general, the performance of queries with filters, joins, and aggregations
on DATE/TIMESTAMP columns may change due to underlying changes of the query
plan. Nevertheless, the changes here lead to better estimations so it is
considered a bug fix rather than behavior change.
> Missing NDV and HLL stats for DATE/TIMESTAMP columns during optimization
> ------------------------------------------------------------------------
>
> Key: HIVE-29534
> URL: https://issues.apache.org/jira/browse/HIVE-29534
> Project: Hive
> Issue Type: Bug
> Reporter: Konstantin Bereznyakov
> Assignee: Konstantin Bereznyakov
> Priority: Major
> Labels: pull-request-available
> Attachments: 6406.patch
>
>
> StatsUtils#getColStatistics method does not fetch/update NDV and HLL stats
> for DATE/TIMESTAMP column types from the metastore. As a result, the NDV/HLL
> statistics for such columns is either zero or empty leading to sub-optimal
> query plans.
> [https://github.com/apache/hive/blob/bbd83dff5bfc8b8ce018476391469da3331216dd/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L840]
> [https://github.com/apache/hive/blob/bbd83dff5bfc8b8ce018476391469da3331216dd/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L870]
> Adding this info seems to change the output of about 100 .out files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
