abstractdog commented on code in PR #6805:
URL: https://github.com/apache/hive/pull/6805#discussion_r4388895244
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java:
##########
@@ -258,19 +294,20 @@ public void load(MapJoinTableContainer[] mapJoinTables,
} catch (Exception e) {
LOG.debug("Failed to get value for counter
APPROXIMATE_INPUT_RECORDS", e);
}
- long keyCount = Math.max(estKeyCount, inputRecords);
+ long keyCount = keyCountForSizing(estKeyCount, inputRecords);
initHTLoadingService(keyCount);
Review Comment:
Suggestion: pick the thread count from the larger signal, and keep
`keyCountForSizing` (min) for sizing the table:
```java
long keyCount = keyCountForSizing(estKeyCount, inputRecords);
initHTLoadingService(Math.max(estKeyCount, inputRecords));
```
The thread count doesn't change how much memory is allocated up front. The
container splits the min-based capacity across power-of-two partitions, so the
total slot-array bytes stay the same (as the Reducer 9 note says). The risk
runs the other way from sizing:
- **Too many threads** on a table that really is small: a few tiny
partitions instead of one, a negligible cost.
- **Too few threads** when the optimizer underestimates a large table: the
whole load runs on one thread, and the cost grows with the real size. That's
the case HIVE-23953 added the counter for, and with TEZ-4757 fixed the counter
is more likely to be the accurate signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
