[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7710/ -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 6 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 17 Dec 2021 18:33:53 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9943/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 6 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 17 Dec 2021 12:37:53 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7710/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 6 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 17 Dec 2021 12:19:19 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/18083/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/18083/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@886 PS6, Line 886: * Reloads partitions from the event if they exist. Does not fetch those partitions from HMS. line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/18083/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/18083/6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4242 PS6, Line 4242:* @param fileMetadataLoadOpts describes how to reload file metadata for the partsFromEvent line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 6 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 17 Dec 2021 12:14:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18083 to look at the new patch set (#6). Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor HdfsPartition in catalogD is a collection of files and each file is represented by a FileDescriptor. When the event processor processes an AlterPartitionEvent, currently it reloads partition metadata along with its file metadata. Reloading of file metadata is a relatively expensive operation as it involves listing files in the underlying filesystem. This patch makes an improvement in processing alter partition event for external tables. The proposed solution is to compare sd of HMS partition object received from AlterPartition event with sd of already cached HdfsPartition. Reload file metadata if any of the following is true * HdfsParition's sd does not match sd of HMS partition object * Cache directive is present for HdfsPartition. File metadata reloading is done to fetch latest file blocks info. Testing: * Added new unit tests Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 --- A fe/src/main/java/org/apache/impala/catalog/FileMetadataLoadOpts.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 315 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/6 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 6 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7709/ -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 5 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 Dec 2021 19:47:13 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9938/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 5 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 Dec 2021 13:50:28 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18083 to look at the new patch set (#5). Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor HdfsPartition in catalogD is a collection of files and each file is represented by a FileDescriptor. When the event processor processes an AlterPartitionEvent, currently it reloads partition metadata along with its file metadata. Reloading of file metadata is a relatively expensive operation as it involves listing files in the underlying filesystem. This patch makes an improvement in processing alter partition event for external tables. The proposed solution is to compare HMS partition object received from AlterPartition event and compare its storage descriptor with the storage descriptor of already cached HdfsPartition. Reload file metadata if any of the following is true * HdfsParition's sd does not match sd of HMS partition object * Cache directive is present for HdfsPartition. File metadata reloading is done to fetch latest file blocks info. Testing: * Added new unit tests Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 --- A fe/src/main/java/org/apache/impala/catalog/FileMetadataLoadOpts.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 243 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/5 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 5 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7709/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 5 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 Dec 2021 13:30:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 Dec 2021 13:15:41 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/18083/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/18083/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@2877 PS4, Line 2877: List partitionsToDrop = tbl.getPartitionsForNames(Arrays.asList(partName)); line too long (94 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 16 Dec 2021 12:53:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18083 to look at the new patch set (#4). Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/service/FileMetadataLoadOpts.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 245 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/4 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9927/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 14 Dec 2021 18:45:52 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18083 to look at the new patch set (#2). Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/service/FileMetadataLoadOpts.java 3 files changed, 76 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/2 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/18083/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/18083/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2698 PS2, Line 2698: return reloadPartitionsFromNames(client, partNames, reason, FileMetadataLoadOpts.LOAD); line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 14 Dec 2021 18:23:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9905/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 1 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 10 Dec 2021 13:13:21 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Sourabh Goyal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18083 Change subject: [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. [WIP] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 1 file changed, 17 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/1 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 1 Gerrit-Owner: Sourabh Goyal
