Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16363 )
Change subject: IMPALA-4364: [Addendum] Compare specific fields in StorageDescriptor ...................................................................... IMPALA-4364: [Addendum] Compare specific fields in StorageDescriptor The query option REFRESH_UPDATED_HMS_PARTITIONS was introduced earlier in IMPALA-4364 to detect changes in the partition objects in HMS when a refresh table command is issued. Originally, it relied on using the StorageDescriptor#equals() method to determine if the Partition in catalogd is same as partition in HMS with while executing the refresh statement. However, using StorageDescriptor#equals() is dependent on HMS version and may introduce inconsistent behaviors after upgrades. For example, when we backported the original patch to older distribution which uses Hive-2, the SkewedInfo field of StorageDescriptor is not null. This field causes the comparison logic to fail, since catalogd doesn't store the SkewedInfo field in the cached StorageDescriptor to optimize memory usage. This patch modifies the comparison logic to use explicit implementation in HdfsPartition class which compares only some fields which are cached in the HdfsPartition object. Testing: 1. Added a new test for the comparison method. 2. Modified existing test for the query option. Change-Id: I90c797060265f8f508d0b150e15da3d0f9961b9b Reviewed-on: http://gerrit.cloudera.org:8080/16363 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Vihang Karajgaonkar <vih...@cloudera.com> --- M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/test/java/org/apache/impala/catalog/HdfsPartitionSdCompareTest.java M tests/metadata/test_reset_metadata.py 4 files changed, 156 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Vihang Karajgaonkar: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I90c797060265f8f508d0b150e15da3d0f9961b9b Gerrit-Change-Number: 16363 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
