Denys Kuzmenko created HIVE-30157:
-------------------------------------
Summary: Probe decode: the NDV ratio that picks a scan's
probe-decode map join looks up both keys in the wrong statistics
Key: HIVE-30157
URL: https://issues.apache.org/jira/browse/HIVE-30157
Project: Hive
Issue Type: Bug
Reporter: Denys Kuzmenko
TezCompiler.selectLowestRatioProbeDecodeMapJoin chooses, for each TableScan,
the map join whose hash table the reader can use to skip rows
(probeDecodeDetails). It picks the candidate with the lowest
getProbeDecodeNDVRatio, which is meant to be NDV(hash table key) / NDV(scanned
column). Both lookups use column names from the wrong namespace:
- Big-table key: looked up in the TableScan statistics under its post-Select
name (for example _col1). TableScan statistics are keyed by table column names
(inv_item_sk), so whenever a Select sits between the scan and the map join, the
lookup misses and the ratio falls back to the scan's row count.
- Small-table key: looked up in the map join's output statistics. The key
belongs to the namespace of the small-side ReduceSink's input. When no output
column has that name, the ratio silently becomes join output rows / scan rows.
When an output column happens to share the name, the ratio uses that unrelated
column's post-join NDV.
bigKeyColName itself is resolved correctly, by tracing the key back to its
TableScan column, but only for the winner and only after the ratio has chosen
it.
Evidence in master's goldens:
- probedecode_mapjoin_stats.q.out: keyRatio:0.16666666666666666. The intended
ratio is NDV(seller_dim.key4)=2 / NDV(orders_fact.key3)=6 = 0.333.
- tpcds30tb query37/query82: keyRatio × scan rows = exactly 1.0, and query21:
4. Each is an NDV of 1 or 4 from an unrelated post-join column.
- 61 of the 507 probeDecodeDetails lines have keyRatio > 1.0.
- Example with a plan change: in TPC-DS q39, an inserted GROUP BY renames a
column so a lookup that used to miss now hits. The target flips from
inv_date_sk (ratio 0.017) to inv_item_sk (7.6e-10 = 1/1,311,525,000).
*Impact*: no reader filters on the probe context yet; today it is consumed only
by a log line in OrcEncodedDataConsumer. Results and performance are unaffected
now. Once a reader (ORC, or Parquet through the LLAP cache) applies probe
decode, this ratio decides which hash table filters the scan, and a wrong
winner probes every row for nothing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
