kasakrisz opened a new pull request, #6963:
URL: https://github.com/apache/hive/pull/6963
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA:
https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a
faster review.
-->
### What changes were proposed in this pull request?
* Add a new Tez physical optimization: PartialGroupByJoinTransposeOptimizer.
It pushes a copy of the map-side (hash) GROUP BY below an inner join. The copy
goes to the join input that holds every aggregation argument.
```
TS - SEL - RS \
JOIN - SEL - GBY(hash) - RS - GBY(mergepartial)
TS - SEL - RS /
=>
TS - SEL - GBY(partialhash) - RS \
JOIN - GBY(hash) - RS - GBY(mergepartial)
TS - SEL -------------------- RS /
```
* The pushed GROUP BY runs in a new mode: partialhash, in the vertex of the
join input, before its existing shuffle.
* The GROUP BY above the join keeps its mode and output, but it merges the
partial results (PARTIAL2) instead of reading the rows. The reduce-side GROUP
BY does not change.
* When a pushed GROUP BY that is matched again can be pushed further down a
chain of joins and a Select operator that projects its keys and partial results
replaces it above the lower join.
* The pushed GROUB BY keys are the original keys available on that input,
the input's join keys, and its columns used by residual (non-equi) join filters.
* The new optimization is run by TezCompiler after annotates the statistics.
It is run before the TopNKey optimization and dynamic partition pruning.
Otherwise, TopNKey operators and semijoin branches would block the push-down.
* Synthetic join predicates of residual filters are re-pointed to the output
of the pushed GROUP BY.
* In StatsRulesProcFactory a GROUP BY with partial hash mode is estimated at
its parent's row count. This is pessimistic, so map join conversion decisions
don't change.
* Synthetic join predicates doesn't change filter statistics. This was
necessary because stats annotation run is moved before dynamic partition
pruning which removes Synthetic join predicates.
### Why are the changes needed?
To improve performance of queries with joins and aggregates.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=partial_aggr_join_transpose.q -pl itests/qtest -Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]