Alex Behm has posted comments on this change. Change subject: IMPALA-4172/IMPALA-3653: Improvements to block metadata loading ......................................................................
Patch Set 12: Code-Review+1 (7 comments) I'm pretty happy with the changes. Let's let Dimitris take a look. http://gerrit.cloudera.org:8080/#/c/5148/12//COMMIT_MSG Commit Message: Line 15: We loop throuh each and every file in the table/partition directories through Line 17: This results in large no. of RPC calls to namenode, especially with no. -> number to the NameNode especially for large tables Line 35: mention the behavior change of REFRESH http://gerrit.cloudera.org:8080/#/c/5148/12/fe/src/main/java/org/apache/impala/catalog/DiskIdMapper.java File fe/src/main/java/org/apache/impala/catalog/DiskIdMapper.java: Line 31: * - To maintain consistent mapping across all the table instances so that the disk thread ... a consistent mapping ... so that the assignment of scan ranges to I/O threads is balanced and consistent for all scans on the same host. http://gerrit.cloudera.org:8080/#/c/5148/12/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: Line 288: * Queries the filesystem to load the file block metadata (e.g. DFS blocks) for the Suggest rephrasing: Drops and re-loads the block metadata for all partitions in 'partsByPath' whose location is under the given 'dirPath'. It involves the following steps: ... Line 296: * and enumerate all its blocks and their corresponding hosts and disk IDs. remove part about disk ids, I think that's the next point Line 778: LOG.debug("partsByPath size: " + partsByPath.size()); check log lvl -- To view, visit http://gerrit.cloudera.org:8080/5148 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ie127658172e6e70dae441374530674a4ac9d5d26 Gerrit-PatchSet: 12 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-HasComments: Yes
