Misha Dmitriev has posted comments on this change. ( http://gerrit.cloudera.org:8080/10982 )
Change subject: IMPALA-7219. 7.5% of Catalog Server heap wasted by empty HashMaps and ArrayLists ...................................................................... Patch Set 2: Thank you for your comments/clarifications, guys. I see what you mean now. I've checked the heap dump more carefully, and now I realize that all the StructType instances (all of which have both of their data fields, fieldMap_ and fields_, empty) come from IncompleteTable's. Here is the relevant excerpt from the jxray heap report: 182,587K (1.2%): org.apache.impala.catalog.StructType: 7790403 / 100% objects ↖org.apache.impala.catalog.ArrayType.itemType_ ↖org.apache.impala.catalog.IncompleteTable.type_ ↖{java.util.concurrent.ConcurrentHashMap}.values ↖org.apache.impala.catalog.CatalogObjectCache.metadataCache_ ↖org.apache.impala.catalog.Db.tableCache_ ↖{java.util.concurrent.ConcurrentHashMap}.values ↖java.util.concurrent.atomic.AtomicReference.value ↖org.apache.impala.catalog.CatalogServiceCatalog.dbCache_ ↖org.apache.impala.catalog.BuiltinsDb.parentCatalog_ ↖Java Static org.apache.impala.catalog.Catalog.builtinsDb_ So, StructType objects themselves take 1.2% of the heap, plus their empty fieldMap_ maps take 2.5% and empty fields_ lists take 1.2%. And looks like if we simply initialize these StructType objects lazily (more precisely, the 'ArrayType type_' field in Table), we will save all the above memory, i.e. considerably more than the current patch saves. Furthermore, the lazy changes in StructType itself can be reverted, which would keep the code more readable. Makes sense? I think this is the best I can do in Impala code with my current very limited understanding of it. More advanced changes, like making IncompleteTable not inherit from Table, would likely require a much deeper knowledge of how this code works. -- To view, visit http://gerrit.cloudera.org:8080/10982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6 Gerrit-Change-Number: 10982 Gerrit-PatchSet: 2 Gerrit-Owner: Misha Dmitriev <count...@gmail.com> Gerrit-Reviewer: Misha Dmitriev <count...@gmail.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Comment-Date: Fri, 20 Jul 2018 19:50:32 +0000 Gerrit-HasComments: No