Bharath Vissapragada has uploaded a new patch set (#2). Change subject: IMPALA-4943: Speed up block md loading for add/recover partition calls. ......................................................................
IMPALA-4943: Speed up block md loading for add/recover partition calls. This change makes alter table add/recover partitions calls use the per directory block metadata loading routines instead of doing it per file. This is done since these calls always load the entire partition directory from scratch and there is no advantage in loading them incrementally on a per-file basis. Tests: Ran core tests and the metadata benchmark tests. (I) Improvement: METADATA-BENCHMARKS() 100K-PARTITIONS-1M-FILES-03-RECOVER [text / none / none] (718.62s -> 549.91s [-23.48%]) (I) Improvement: METADATA-BENCHMARKS() 100K-PARTITIONS-1M-FILES-08-ADD-PARTITION [text / none / none] (46.92s -> 26.20s [-44.15%]) Change-Id: I331f1f090518f317bcd7df069e480edbd8f039f1 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 1 file changed, 21 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/6651/2 -- To view, visit http://gerrit.cloudera.org:8080/6651 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331f1f090518f317bcd7df069e480edbd8f039f1 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com>
