Krisztian Kasa created HIVE-30208:
-------------------------------------
Summary: Estimate multi-key join cardinality with a configurable
key independence factor
Key: HIVE-30208
URL: https://issues.apache.org/jira/browse/HIVE-30208
Project: Hive
Issue Type: Improvement
Components: CBO
Reporter: Krisztian Kasa
When a join has several equi-join keys, CBO (HiveRelMdSelectivity) and the
physical planner (StatsRulesProcFactory.JoinStatsRule) have to combine the
keys' NDVs (number of distinct values) into the NDV of the composite key. Today
there are only two options, chosen by a boolean setting:
hive.cbo.stats.correlated.multi.key.joins /
hive.stats.correlated.multi.key.joins
- = true (default): keys are assumed fully correlated and only the largest NDV
is used.
- = false: exponential back-off: ndv1 * ndv2^(1/2) * ndv3^(1/4) *...
The default "fully correlated" estimate ignores every key except one, so the
denominator is too small and multi-key joins are heavily overestimated.
Example: on a 10TB TPC_DS dataset TPC-DS q29
(store_sales x store_returns on customer, item and ticket number): 748.6M rows
estimated vs 11.0M actual.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
