konstantinb commented on code in PR #6505:
URL: https://github.com/apache/hive/pull/6505#discussion_r4475444391
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java:
##########
@@ -1820,9 +1835,24 @@ private boolean checkMapSideAggregation(GroupByOperator
gop,
// estimate size of key from column statistics
long avgKeySize = 0;
+ // lazily computed on first unknown NDV (null = not yet looked up)
+ Long parentNumRows = null;
for (ColStatistics cs : colStats) {
if (cs != null) {
- numEstimatedRows = StatsUtils.safeMult(numEstimatedRows,
cs.getCountDistint());
+ long ndv = cs.getCountDistint();
+ if (ndv < 0) {
+ if (parentNumRows == null) {
+ // unknown NDV: fall back to parentNumRows / 2, matching the
heuristic
+ // used elsewhere in this file when GROUP BY cardinality
cannot be computed
+ Statistics parentStats =
gop.getParentOperators().get(0).getStatistics();
+ parentNumRows = (parentStats != null) ?
parentStats.getNumRows() : -1L;
+ }
+ if (parentNumRows <= 0) {
+ return false;
+ }
+ ndv = parentNumRows / 2;
+ }
+ numEstimatedRows = StatsUtils.safeMult(numEstimatedRows, ndv);
Review Comment:
Agreed: this changed master's decision, including for a single unknown key
on a large input. Fixed in
[d69875a](https://github.com/apache/hive/pull/6505/commits/d69875aefc): an
unknown NDV now keeps master's outcome (unknown was stored as 0, so the
estimated hash table was empty), and the known-NDV product, which predates this
PR, is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
