[
https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294271#comment-15294271
]
Gopal V commented on HIVE-13809:
--------------------------------
{{1,600,000,000}} doesn't look like a good scenario to fix around for a
map-join.
Back of the envelope, 1.6 billion keys in a 2Gb hashtable comes down to <2
bytes per key, which is obvious going to OOM there, even divided by 16 (i.e 32
bytes per key+value).
I suspect the root issue has more to do with the statistics involved here,
which might be completely bogus.
Ideally, we should be capping the bloom filter estimates at 4 million keys,
which is nearly the probe limit of useful hashtables - any real data-set bigger
than 4M, the total number of rehashes at 2Gb will also OOM the hashtables.
So in this scenario, I'm nearly convinced that the 1.6 billion number is bogus
or bad planning.
> Hybrid Grace Hash Join memory usage estimation didn't take into account the
> bloom filter size
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-13809
> URL: https://issues.apache.org/jira/browse/HIVE-13809
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.0.0, 2.1.0
> Reporter: Wei Zheng
> Assignee: Wei Zheng
>
> Memory estimation is important during hash table loading, because we need to
> make the decision of whether to load the next hash partition in memory or
> spill it. If the assumption is there's enough memory but it turns out not the
> case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into
> account the bloom filter size. In large test cases (TB scale) the bloom
> filter grows as big as hundreds of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
