[
https://issues.apache.org/jira/browse/HIVE-30183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-30183:
----------------------------------
Labels: pull-request-available (was: )
> LLAP: a reduce-side merge join skips the minimum reducers per executor
> ----------------------------------------------------------------------
>
> Key: HIVE-30183
> URL: https://issues.apache.org/jira/browse/HIVE-30183
> Project: Hive
> Issue Type: Bug
> Components: llap, Query Planning
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
>
> LlapDecider raises the minimum reducers of an auto-reduced vertex to
> hive.tez.llap.min.reducer.per.executor (0.33) x the cluster's executors
> (adjustAutoParallelism), so auto-reduce does not shrink a reducer to a
> handful of tasks on a large cluster. It applies the floor only to a
> ReduceWork: a reduce-side merge join is a MergeJoinWork, whose main and
> merged reduce works feed one vertex through their edges, and is skipped.
> TPC-DS q51 at 10 TB on 24 LLAP daemons (288 executors): the FULL OUTER merge
> join vertex ran 13 tasks while its input reducers and the PTF after it ran
> 253; the floor would have kept it at 96 or more.
> Fix: apply the floor to the merge join's main work and each merged reduce
> work (their edge properties are the ones the vertex's edges use).
> TestLlapDecider covers it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
