[Impala-ASF-CR] IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap.
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/10982 ) Change subject: IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap. .. Abandoned Appears stale, reopen if you disagree -- To view, visit http://gerrit.cloudera.org:8080/10982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6 Gerrit-Change-Number: 10982 Gerrit-PatchSet: 8 Gerrit-Owner: Misha Dmitriev Gerrit-Reviewer: Misha Dmitriev Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10982 ) Change subject: IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap. .. Patch Set 4: I0721 20:07:39.425611 85266 jni-util.cc:230] org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: tpch.lineitem at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1290) at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1217) at org.apache.impala.catalog.TableLoader.load(TableLoader.java:80) at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:241) at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:238) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.impala.catalog.Table.clearColumns(Table.java:192) at org.apache.impala.catalog.HdfsTable.loadSchema(HdfsTable.java:1583) at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1257) ... 8 more I0721 20:07:39.426174 85266 status.cc:125] TableLoadingException: Failed to load metadata for table: tpch.lineitem CAUSED BY: NullPointerException: null Logs from the data loading process are here: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2771/artifact/Impala/logs_static/logs/data_loading/ -- To view, visit http://gerrit.cloudera.org:8080/10982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6 Gerrit-Change-Number: 10982 Gerrit-PatchSet: 4 Gerrit-Owner: Misha Dmitriev Gerrit-Reviewer: Misha Dmitriev Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sun, 22 Jul 2018 00:15:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap.
Misha Dmitriev has posted comments on this change. ( http://gerrit.cloudera.org:8080/10982 ) Change subject: IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap. .. Patch Set 4: Patch set 4 failed twice in the Jenkins build, so I suspect there is some problem in it. The last failure is https://jenkins.impala.io/job/pre-review-test/190/ Unfortunately, I don't see any clues in the main build log, and I don't know how to get to the Java impala catalog logs. Any help would be appreciated. -- To view, visit http://gerrit.cloudera.org:8080/10982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6 Gerrit-Change-Number: 10982 Gerrit-PatchSet: 4 Gerrit-Owner: Misha Dmitriev Gerrit-Reviewer: Misha Dmitriev Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 21 Jul 2018 21:36:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap.
Misha Dmitriev has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10982 ) Change subject: IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap. .. IMPALA-7219. Avoid wasting ~160 bytes per IncompleteTable in Catalog Server heap. This change switches initialization from eager to lazy for the following fields in Table: HashMap colsByName_ and ArrayType type_. In this way, we avoid wasting memory when HashMaps stay empty (an empty HashMap still uses at least 48 bytes in the heap), and when a StructType with its own empty HashMap and ArrayList instances is initialized, but stays unused. I have calculated ArrayType, StructType, two HashMaps and one ArrayList collectively use up 160 bytes in the heap per a Table. Further analysis can be found in the JIRA. This optimization becomes really relevant when a catalog server loads a very large number (millions) of tables, and all/most of them are represented as IncompleteTable instances. Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6 --- M fe/src/main/java/org/apache/impala/catalog/Table.java 1 file changed, 24 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/10982/4 -- To view, visit http://gerrit.cloudera.org:8080/10982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If9c75f65ecb3ba3f2c739fa483a84dc052f471c6 Gerrit-Change-Number: 10982 Gerrit-PatchSet: 4 Gerrit-Owner: Misha Dmitriev Gerrit-Reviewer: Misha Dmitriev Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
