[
https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294166#comment-15294166
]
Gopal V commented on HIVE-13809:
--------------------------------
[~wzheng]: before we actually subtract such a large amount of memory from the
join algorithms, maybe we should figure out the input parameters to the bloom
filter creation.
The reason it wasn't accounted for strictly was because of the relatively small
size of bloom filters - a 4 million keyset with 0.5% false positive rate should
result in a bloom filter which is approx ~6Mb.
Would be a good idea to figure out the false positive rate + estimated key
count of the bloom filter which grew to hundreds of Mbs and see if there's an
implementation issue there.
> Hybrid Grace Hash Join memory usage estimation didn't take into account the
> bloom filter size
> ---------------------------------------------------------------------------------------------
>
> Key: HIVE-13809
> URL: https://issues.apache.org/jira/browse/HIVE-13809
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.0.0, 2.1.0
> Reporter: Wei Zheng
> Assignee: Wei Zheng
>
> Memory estimation is important during hash table loading, because we need to
> make the decision of whether to load the next hash partition in memory or
> spill it. If the assumption is there's enough memory but it turns out not the
> case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into
> account the bloom filter size. In large test cases (TB scale) the bloom
> filter grows as big as hundreds of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
