Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10450 )
Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations. ...................................................................... Patch Set 2: (2 comments) Just interested in this optimization. May I ask some questions? Looks like we are optimizing the case when partitionsToUpdate != null and partitions were neither dropped, created. Can we optimize the case that partitionsToUpdate != null and some partitions are dropped? For example when an INSERT OVERWRITE statement updates the majority of the partitions and only drops few of them. http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1409 PS2, Line 1409: size() == 0 nit: can be simplified by isEmpty() http://gerrit.cloudera.org:8080/#/c/10450/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1410 PS2, Line 1410: partitionsToUpdateFileMdByPath = getPartitionsByPath(partitionsToUpdate); : loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true); Looks like the original codes perform the same as these two lines. Since dropPartitions, loadPartitionsFromMetastore and loadPartitionsFromMetastore will return fast when their first parameter is empty. partitionExist is not empty means partitionsToUpdate != null, so the original codes will perform as these two lines. My question is what are the HMS requests we save? -- To view, visit http://gerrit.cloudera.org:8080/10450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaabdf38af3f30c65ada9734eb471dbfa6ecdd74a Gerrit-Change-Number: 10450 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Sat, 19 May 2018 13:17:41 +0000 Gerrit-HasComments: Yes
