[
https://issues.apache.org/jira/browse/HIVE-17304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124382#comment-16124382
]
Prasanth Jayachandran commented on HIVE-17304:
----------------------------------------------
This provides a better estimates and also the estimates are pretty close the
actual object size (observed this from heapdumps) atleast for vectorized case.
Also bringing down the inflation factor from 2.0 to 1.5 as a result. Still
testing this patch on larger dataset.
> ThreadMXBean based memory allocation monitory for hash table loader
> -------------------------------------------------------------------
>
> Key: HIVE-17304
> URL: https://issues.apache.org/jira/browse/HIVE-17304
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Attachments: HIVE-17304.1.patch
>
>
> Hash table memory monitoring is based on java data model which can be
> unreliable because of various reasons (wrong object size estimation, adding
> new variables to any class without accounting its size for memory monitoring,
> etc.). We can use allocation size per thread that is provided by ThreadMXBean
> and fallback to DataModel in case if JDK doesn't support thread based
> allocations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
