[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Reviewed-on: http://gerrit.cloudera.org:8080/18324 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 31 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 05:50:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 4: Code-Review+2 LGTM. Thanks for working on this! -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 01:22:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7988/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 01:23:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 31 Mar 2022 01:23:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10362/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 30 Mar 2022 16:38:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has posted comments on this change. (
http://gerrit.cloudera.org:8080/18324 )
Change subject: IMPALA-11181: Improving performance of compaction checking
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@705
PS3, Line 705: if
(partNameToCompactionId.containsKey(entry.getKey().getName())) {
: stalePartitions.add(entry.getKey());
: iter.remove();
> nit: Can we optimize this to the following case?
Done
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@295
PS3, Line 295:
> nit: Could you add a blank line before this?
Done
--
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 4
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Wed, 30 Mar 2022 16:20:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 31 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/4 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18324 )
Change subject: IMPALA-11181: Improving performance of compaction checking
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@705
PS3, Line 705: long latestCompactionId =
partNameToCompactionId.getOrDefault(
: entry.getKey().getName(), -1L);
: if (latestCompactionId > 0) {
nit: Can we optimize this to the following case?
if (partNameToCompactionId.containsKey(entry.getKey().getName()))
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/18324/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@295
PS3, Line 295: // The last committed compaction id in the table level. It
will be sent as a filter to
nit: Could you add a blank line before this?
--
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 3
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Wed, 30 Mar 2022 02:04:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 3: Code-Review+1 @Yu-Wen: Thanks for addressing all my comments. Patch looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 29 Mar 2022 23:02:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 29 Mar 2022 09:39:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10351/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 29 Mar 2022 05:10:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7978/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 29 Mar 2022 04:50:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 30 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/3 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Sourabh Goyal has posted comments on this change. (
http://gerrit.cloudera.org:8080/18324 )
Change subject: IMPALA-11181: Improving performance of compaction checking
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@297
PS2, Line 297: private long lastCompactionId_ = -1;
Please add some more description on the use of this variable, how it is
supposed to get updated etc
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@974
PS2, Line 974: if (partition.getLastCompactionId() > 0) {
nit: we can skip the check since the following condition is sufficient:
lastCompactionId_ = Math.max(lastCompactionId_,
partition.getLastCompactionId());
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@485
PS2, Line 485: // Compaction id is not provided with DirectMetaProvider
implementation
Why are we changing the old behavior?
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@538
PS2, Line 538: if (metas.isEmpty()) {
nit: We can skip this check since checkLatestCompaction() already checks metas
size.
http://gerrit.cloudera.org:8080/#/c/18324/2/fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java@543
PS2, Line 543: List stalePartitions =
MetastoreShim.checkLatestCompaction(
nit: if we want to log the time that checkLatestCompaction() takes, it would be
better to move the Stopwatch check in MetastoreShim.checkLatestCompaction()
--
To view, visit http://gerrit.cloudera.org:8080/18324
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c
Gerrit-Change-Number: 18324
Gerrit-PatchSet: 2
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Fri, 25 Mar 2022 00:16:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 23 Mar 2022 00:11:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7955/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 22 Mar 2022 19:22:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7950/ -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 22 Mar 2022 02:33:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10317/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 21 Mar 2022 21:56:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7950/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Mar 2022 21:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 29 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/2 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 2 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7933/ -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Mar 2022 08:10:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10294/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Mar 2022 03:34:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18324 ) Change subject: IMPALA-11181: Improving performance of compaction checking .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7933/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Mar 2022 03:15:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11181: Improving performance of compaction checking
Yu-Wen Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18324 Change subject: IMPALA-11181: Improving performance of compaction checking .. IMPALA-11181: Improving performance of compaction checking After HIVE-25753, we don't need to explicitly set all partitions' name to get the latest compaction id. Besides, we can also send the last compaction id over to HMS so that HMS will send back compaction info only if there are newer compactions. In this way, we can avoid unnecessary data transmitted between HMS and Catalogd. Testing: existing tests Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java 3 files changed, 24 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/18324/1 -- To view, visit http://gerrit.cloudera.org:8080/18324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I32e30ec418ad09bef862e61163539a910c96c44c Gerrit-Change-Number: 18324 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai
