Misha Dmitriev has posted comments on this change. (
http://gerrit.cloudera.org:8080/10982 )
Change subject: IMPALA-7219. 7.5% of Catalog Server heap wasted by empty
HashMaps and ArrayLists
......................................................................
Patch Set 2:
Thank you for your comments/clarifications, guys. I see what you mean now.
I've checked the heap dump more carefully, and now I realize that all the
StructType instances (all of which have both of their data fields, fieldMap_
and fields_, empty) come from IncompleteTable's. Here is the relevant excerpt
from the jxray heap report:
182,587K (1.2%): org.apache.impala.catalog.StructType: 7790403 / 100% objects
↖org.apache.impala.catalog.ArrayType.itemType_
↖org.apache.impala.catalog.IncompleteTable.type_
↖{java.util.concurrent.ConcurrentHashMap}.values
↖org.apache.impala.catalog.CatalogObjectCache.metadataCache_
↖org.apache.impala.catalog.Db.tableCache_
↖{java.util.concurrent.ConcurrentHashMap}.values
↖java.util.concurrent.atomic.AtomicReference.value
↖org.apache.impala.catalog.CatalogServiceCatalog.dbCache_
↖org.apache.impala.catalog.BuiltinsDb.parentCatalog_
↖Java Static org.apache.impala.catalog.Catalog.builtinsDb_
So, StructType objects themselves take 1.2% of the heap, plus their empty
fieldMap_ maps take 2.5% and empty fields_ lists take 1.2%. And looks like if
we simply initialize these StructType objects lazily (more precisely, the
'ArrayType type_' field in Table), we will save all the above memory, i.e.
considerably more than the current patch saves. Furthermore, the lazy changes
in StructType itself can be reverted, which would keep the code more readable.
Makes sense? I think this is the best I can do in Impala code with my current
very limited understanding of it. More advanced changes, like making
IncompleteTable not inherit from Table, would likely require a much deeper
knowledge of how this code works.
--
To view, visit http://gerrit.cloudera.org:8080/10982
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6
Gerrit-Change-Number: 10982
Gerrit-PatchSet: 2
Gerrit-Owner: Misha Dmitriev <count...@gmail.com>
Gerrit-Reviewer: Misha Dmitriev <count...@gmail.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Jul 2018 19:50:32 +0000
Gerrit-HasComments: No
