[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables When applying ALTER_PARTITION events on transactional tables, we refresh the partition using the metadata in events if hms_event_incremental_refresh_transactional_table is enabled (which is the default). This could be wrong if the ALTER_PARTITION event is stale. The partition metadata will be rolled back to a stale state. This patch compares the eventId with the createEventId of the table and ignores those ALTER_PARTITION events that have older (smaller) event ids. Note that we already do this for many other event types, ALTER_PARTITION is somehow missing the checks. Eventually we should depend on the lastSyncedEventId and replace createEventId with it. The self-event detection can also be replaced since self-events are also stale events. These will be addressed in IMPALA-10976. Tests - Verified locally with local-catalog mode and event-processor enabled and iterated test_acid_compute_stats for 1400 times. Without the fix, the test would fail in tens of runs. Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Reviewed-on: http://gerrit.cloudera.org:8080/19020 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 5 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Sep 2022 04:20:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19020/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19020/1//COMMIT_MSG@25 PS1, Line 25: Tests > Thanks Quanlong for catching this. I agree with Csaba that we should add mo That's a good point. Thank Csaba and Yu-Wen! -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 20 Sep 2022 23:13:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 20 Sep 2022 23:13:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8593/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 20 Sep 2022 23:13:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/19020/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19020/1//COMMIT_MSG@25 PS1, Line 25: Tests > The solution looks good, but one thing bugs me: shouldn't the original bug Thanks Quanlong for catching this. I agree with Csaba that we should add more tests around event processing. I just created a follow-up Jira IMPALA-11598. -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 20 Sep 2022 17:10:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/19020/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19020/1//COMMIT_MSG@25 PS1, Line 25: Tests The solution looks good, but one thing bugs me: shouldn't the original bug lead to other issues than the incremental stats one? Maybe it would make sense to add other regression tests where the table is recreated with the same name, and this is done in a loop several time, not just once to avoid flakyness. This should have been caught by an event processing test instead of one about ACID + COMPUTE STATS. I am ok with a follow up Jira instead of adding more tests in this one. -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 20 Sep 2022 04:53:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19020 ) Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 20 Sep 2022 04:45:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11160: Ignore stale ALTER PARTITION events on transactional tables
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19020 Change subject: IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables .. IMPALA-11160: Ignore stale ALTER_PARTITION events on transactional tables When applying ALTER_PARTITION events on transactional tables, we refresh the partition using the metadata in events if hms_event_incremental_refresh_transactional_table is enabled (which is the default). This could be wrong if the ALTER_PARTITION event is stale. The partition metadata will be rolled back to a stale state. This patch compares the eventId with the createEventId of the table and ignores those ALTER_PARTITION events that have older (smaller) event ids. Note that we already do this for many other event types, ALTER_PARTITION is somehow missing the checks. Eventually we should depend on the lastSyncedEventId and replace createEventId with it. The self-event detection can also be replaced since self-events are also stale events. These will be addressed in IMPALA-10976. Tests - Verified locally with local-catalog mode and event-processor enabled and iterated test_acid_compute_stats for 1400 times. Without the fix, the test would fail in tens of runs. Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/19020/1 -- To view, visit http://gerrit.cloudera.org:8080/19020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5bb8cfc213093f3bbd0359c7084b277a3bd5264a Gerrit-Change-Number: 19020 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
