Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/11227 )
Change subject: IMPALA-7047. Refreshing partitions should not make an RPC per file ...................................................................... Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/11227/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11227/4//COMMIT_MSG@12 PS4, Line 12: since the prior load, it was instead calling it on every file. General question. In HDFS, we can tell if a file was added or removed. Once written, HDFS files are immutable, so we don't have to worry about any file "changing." Still, HDFS could rebalance blocks. In the present code, we have no way to know that block assignments changed and will retain stale block info in the cache. Do we know if HDFS migrates blocks in any of our supported configurations? Since stale blocks stay in the cache, do we have a way in the query to detect stale blocks and fetch new block info? Say, at scan time? -- To view, visit http://gerrit.cloudera.org:8080/11227 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2051b96599206164aaa06ecbdf64374c46eda956 Gerrit-Change-Number: 11227 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Paul Rogers <par0...@yahoo.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com> Gerrit-Comment-Date: Tue, 27 Nov 2018 06:14:21 +0000 Gerrit-HasComments: Yes
