[
https://issues.apache.org/jira/browse/HIVE-30157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30157:
----------------------------------
Description:
TezCompiler.selectLowestRatioProbeDecodeMapJoin chooses, for each TableScan,
the map join whose hash table the reader can use to skip rows
(probeDecodeDetails). It picks the candidate with the lowest
getProbeDecodeNDVRatio, which is meant to be NDV(hash table key) / NDV(scanned
column). Both lookups use column names from the wrong namespace:
- Big-table key: looked up in the TableScan statistics under its post-Select
name (for example _col1). TableScan statistics are keyed by table column names
(inv_item_sk), so whenever a Select sits between the scan and the map join, the
lookup misses and the ratio falls back to the scan's row count.
- Small-table key: looked up in the map join's output statistics. The key
belongs to the namespace of the small-side ReduceSink's input. When no output
column has that name, the ratio silently becomes join output rows / scan rows.
When an output column happens to share the name, the ratio uses that unrelated
column's post-join NDV.
bigKeyColName itself is resolved correctly, by tracing the key back to its
TableScan column, but only for the winner and only after the ratio has chosen
it.
*Evidence in master's goldens*:
- probedecode_mapjoin_stats.q.out: keyRatio:0.16666666666666666. The intended
ratio is NDV(seller_dim.key4)=2 / NDV(orders_fact.key3)=6 = 0.333.
- tpcds30tb query37/query82: keyRatio × scan rows = exactly 1.0, and query21:
4. Each is an NDV of 1 or 4 from an unrelated post-join column.
- 61 of the 507 probeDecodeDetails lines have keyRatio > 1.0.
- Example with a plan change: in TPC-DS q39, an inserted GROUP BY renames a
column so a lookup that used to miss now hits. The target flips from
inv_date_sk (ratio 0.017) to inv_item_sk (7.6e-10 = 1/1,311,525,000).
*Impact*: no reader filters on the probe context yet; today it is consumed only
by a log line in OrcEncodedDataConsumer. Results and performance are unaffected
now. Once a reader (ORC, or Parquet through the LLAP cache) applies probe
decode, this ratio decides which hash table filters the scan, and a wrong
winner probes every row for nothing.
was:
TezCompiler.selectLowestRatioProbeDecodeMapJoin chooses, for each TableScan,
the map join whose hash table the reader can use to skip rows
(probeDecodeDetails). It picks the candidate with the lowest
getProbeDecodeNDVRatio, which is meant to be NDV(hash table key) / NDV(scanned
column). Both lookups use column names from the wrong namespace:
- Big-table key: looked up in the TableScan statistics under its post-Select
name (for example _col1). TableScan statistics are keyed by table column names
(inv_item_sk), so whenever a Select sits between the scan and the map join, the
lookup misses and the ratio falls back to the scan's row count.
- Small-table key: looked up in the map join's output statistics. The key
belongs to the namespace of the small-side ReduceSink's input. When no output
column has that name, the ratio silently becomes join output rows / scan rows.
When an output column happens to share the name, the ratio uses that unrelated
column's post-join NDV.
bigKeyColName itself is resolved correctly, by tracing the key back to its
TableScan column, but only for the winner and only after the ratio has chosen
it.
Evidence in master's goldens:
- probedecode_mapjoin_stats.q.out: keyRatio:0.16666666666666666. The intended
ratio is NDV(seller_dim.key4)=2 / NDV(orders_fact.key3)=6 = 0.333.
- tpcds30tb query37/query82: keyRatio × scan rows = exactly 1.0, and query21:
4. Each is an NDV of 1 or 4 from an unrelated post-join column.
- 61 of the 507 probeDecodeDetails lines have keyRatio > 1.0.
- Example with a plan change: in TPC-DS q39, an inserted GROUP BY renames a
column so a lookup that used to miss now hits. The target flips from
inv_date_sk (ratio 0.017) to inv_item_sk (7.6e-10 = 1/1,311,525,000).
*Impact*: no reader filters on the probe context yet; today it is consumed only
by a log line in OrcEncodedDataConsumer. Results and performance are unaffected
now. Once a reader (ORC, or Parquet through the LLAP cache) applies probe
decode, this ratio decides which hash table filters the scan, and a wrong
winner probes every row for nothing.
> Probe decode: the NDV ratio that picks a scan's probe-decode map join looks
> up both keys in the wrong statistics
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30157
> URL: https://issues.apache.org/jira/browse/HIVE-30157
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> TezCompiler.selectLowestRatioProbeDecodeMapJoin chooses, for each TableScan,
> the map join whose hash table the reader can use to skip rows
> (probeDecodeDetails). It picks the candidate with the lowest
> getProbeDecodeNDVRatio, which is meant to be NDV(hash table key) /
> NDV(scanned column). Both lookups use column names from the wrong namespace:
>
>
>
>
> - Big-table key: looked up in the TableScan statistics under its
> post-Select name (for example _col1). TableScan statistics are keyed by table
> column names (inv_item_sk), so whenever a Select sits between the scan and
> the map join, the lookup misses and the ratio falls back to the scan's row
> count.
>
> - Small-table key: looked up in the map join's output statistics. The key
> belongs to the namespace of the small-side ReduceSink's input. When no output
> column has that name, the ratio silently becomes join output rows / scan
> rows. When an output column happens to share the name, the ratio uses that
> unrelated column's post-join NDV.
> bigKeyColName itself is resolved correctly, by tracing the key back to its
> TableScan column, but only for the winner and only after the ratio has chosen
> it.
> *Evidence in master's goldens*:
>
>
> - probedecode_mapjoin_stats.q.out: keyRatio:0.16666666666666666. The
> intended ratio is NDV(seller_dim.key4)=2 / NDV(orders_fact.key3)=6 = 0.333.
>
> - tpcds30tb query37/query82: keyRatio × scan rows = exactly 1.0, and
> query21: 4. Each is an NDV of 1 or 4 from an unrelated post-join column.
>
> - 61 of the 507 probeDecodeDetails lines have keyRatio > 1.0.
>
>
> - Example with a plan change: in TPC-DS q39, an inserted GROUP BY renames a
> column so a lookup that used to miss now hits. The target flips from
> inv_date_sk (ratio 0.017) to inv_item_sk (7.6e-10 = 1/1,311,525,000).
>
>
>
>
>
> *Impact*: no reader filters on the probe context yet; today it is consumed
> only by a log line in OrcEncodedDataConsumer. Results and performance are
> unaffected now. Once a reader (ORC, or Parquet through the LLAP cache)
> applies probe decode, this ratio decides which hash table filters the scan,
> and a wrong winner probes every row for nothing.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
