Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11227 )
Change subject: IMPALA-7047. Refreshing partitions should not make an RPC per file ...................................................................... Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/11227/6/fe/src/test/java/org/apache/impala/catalog/CatalogTest.java File fe/src/test/java/org/apache/impala/catalog/CatalogTest.java: http://gerrit.cloudera.org:8080/#/c/11227/6/fe/src/test/java/org/apache/impala/catalog/CatalogTest.java@392 PS6, Line 392: catalog_.reloadTable(unpartTable); > Looks like you are still working on a patch (IIUC). Will take a look when i I'm done. The key issue is that we have no test case for 1) a partition directory exists, 2) it is empty, and 3) we call refresh stats. That is the only case where we'll do the original full refresh. But, since the old code (which did the full refresh) worked, and we've tested the new, incremental path, we can allow this patch to proceed, and do a separate patch to with the necessary test setup. http://gerrit.cloudera.org:8080/#/c/11227/7/fe/src/test/java/org/apache/impala/catalog/CatalogTest.java File fe/src/test/java/org/apache/impala/catalog/CatalogTest.java: http://gerrit.cloudera.org:8080/#/c/11227/7/fe/src/test/java/org/apache/impala/catalog/CatalogTest.java@347 PS7, Line 347: // - one listLocatedStatus() per partition, to get the file info > Any idea where this is called? Typo. The "list located statuses" counter corresponds to the listStatus() call in HdfsTable.refreshFileMetadata(). -- To view, visit http://gerrit.cloudera.org:8080/11227 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2051b96599206164aaa06ecbdf64374c46eda956 Gerrit-Change-Number: 11227 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Paul Rogers <par0...@yahoo.com> Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Comment-Date: Mon, 03 Dec 2018 22:59:41 +0000 Gerrit-HasComments: Yes
