Hello Quanlong Huang, Vihang Karajgaonkar, Yu-Wen Lai, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18083
to look at the new patch set (#11).
Change subject: IMPALA-11050: Skip filemetadata reloading in processing
AlterPartition event from event processor
......................................................................
IMPALA-11050: Skip filemetadata reloading in processing AlterPartition
event from event processor
HdfsPartition in catalogD is a collection of files and each file is
represented by a FileDescriptor. Currently when processing an
AlterPartitionEvent, EventProcessor reloads partition metadata along
with its file metadata. File metadata reloading is an expensive
operation as it involves listing files in the underlying filesystem.
This patch makes an improvement in alter partition event processing
for external tables. The proposed solution is to compare storage
descriptor of HMS partition object received from AlterPartition event
with sd of already cached HdfsPartition. Reload file metadata if any
of the following conditions hold true:
* HdfsPartition's sd does not match sd of HMS partition object
* Partition does not exist in catalogd cache but is present in HMS
* Cache directive is present for HdfsPartition. File metadata reloading
is required in this case to fetch latest file blocks info.
Testing:
* Added new unit tests
Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2
---
A fe/src/main/java/org/apache/impala/catalog/FileMetadataLoadOpts.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
5 files changed, 314 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/11
--
To view, visit http://gerrit.cloudera.org:8080/18083
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2
Gerrit-Change-Number: 18083
Gerrit-PatchSet: 11
Gerrit-Owner: Sourabh Goyal <soura...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <soura...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
Gerrit-Reviewer: Yu-Wen Lai <yu-wen....@cloudera.com>
