[
https://issues.apache.org/jira/browse/HIVE-30157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30157:
----------------------------------
Affects Version/s: 4.2.1
> Probe decode: the NDV ratio that picks a scan's probe-decode map join looks
> up both keys in the wrong statistics
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30157
> URL: https://issues.apache.org/jira/browse/HIVE-30157
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.2.1
> Reporter: Denys Kuzmenko
> Priority: Major
>
> TezCompiler.selectLowestRatioProbeDecodeMapJoin chooses, for each TableScan,
> the map join whose hash table the reader can use to skip rows
> (probeDecodeDetails). It picks the candidate with the lowest
> getProbeDecodeNDVRatio, which is meant to be NDV(hash table key) /
> NDV(scanned column). Both lookups use column names from the wrong namespace:
>
>
>
>
> - Big-table key: looked up in the TableScan statistics under its
> post-Select name (for example _col1). TableScan statistics are keyed by table
> column names (inv_item_sk), so whenever a Select sits between the scan and
> the map join, the lookup misses and the ratio falls back to the scan's row
> count.
>
> - Small-table key: looked up in the map join's output statistics. The key
> belongs to the namespace of the small-side ReduceSink's input. When no output
> column has that name, the ratio silently becomes join output rows / scan
> rows. When an output column happens to share the name, the ratio uses that
> unrelated column's post-join NDV.
> bigKeyColName itself is resolved correctly, by tracing the key back to its
> TableScan column, but only for the winner and only after the ratio has chosen
> it.
> *Evidence in master's goldens*:
>
>
> - probedecode_mapjoin_stats.q.out: keyRatio:0.16666666666666666. The
> intended ratio is NDV(seller_dim.key4)=2 / NDV(orders_fact.key3)=6 = 0.333.
>
> - tpcds30tb query37/query82: keyRatio × scan rows = exactly 1.0, and
> query21: 4. Each is an NDV of 1 or 4 from an unrelated post-join column.
>
> - 61 of the 507 probeDecodeDetails lines have keyRatio > 1.0.
>
>
> - Example with a plan change: in TPC-DS q39, an inserted GROUP BY renames a
> column so a lookup that used to miss now hits. The target flips from
> inv_date_sk (ratio 0.017) to inv_item_sk (7.6e-10 = 1/1,311,525,000).
>
>
>
>
>
> *Impact*: no reader filters on the probe context yet; today it is consumed
> only by a log line in OrcEncodedDataConsumer. Results and performance are
> unaffected now. Once a reader (ORC, or Parquet through the LLAP cache)
> applies probe decode, this ratio decides which hash table filters the scan,
> and a wrong winner probes every row for nothing.
>
>
>
> *Fix:*
>
>
> - For each candidate, trace the big-table key to its TableScan column
> (OperatorUtils.findTableOriginColExpr). Skip candidates without one, or whose
> type would need a conversion: probe decode can only filter a column the
> reader decodes.
>
>
> - Ratio = NDV of the small-table key in the statistics of the small-side
> ReduceSink's input / NDV of the traced TableScan column.
>
> - Without column statistics, fall back to the small side's row count, the
> hash table size, instead of the join output rows.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
