[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. IMPALA-10735: Fix Iceberg insert when event polling is enabled INSERT INTO Iceberg table is broken when HMS event polling is enabled. This patch disables INSERT event generation for Iceberg tables because Hive replication doesn't support Iceberg tables anyway. Testing: * added e2e test Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Reviewed-on: http://gerrit.cloudera.org:8080/17560 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 26 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 11 Jun 2021 14:17:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 11 Jun 2021 08:26:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7222/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 11 Jun 2021 08:26:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7220/ -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 11 Jun 2021 00:36:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7220/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 17:40:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 17:40:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 17:26:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8883/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 16:07:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 6: (1 comment) Thanks for the comment! http://gerrit.cloudera.org:8080/#/c/17560/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5022 PS5, Line 5022: if (!BackendConfig.INSTANCE.isInsertEventsEnabled()) return false; > I agree with sending events even if isEventProcessingActive() is false, but Yeah, I agree it's probably better to handle it separately. Especially that it turned out there's also a bug that needs to be fixed. Opened IMPALA-10744 for it. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 15:50:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Hello Qifan Chen, Vihang Karajgaonkar, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17560 to look at the new patch set (#6). Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. IMPALA-10735: Fix Iceberg insert when event polling is enabled INSERT INTO Iceberg table is broken when HMS event polling is enabled. This patch disables INSERT event generation for Iceberg tables because Hive replication doesn't support Iceberg tables anyway. Testing: * added e2e test Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 26 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/17560/6 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/17560/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5022 PS5, Line 5022: if (!BackendConfig.INSTANCE.isInsertEventsEnabled()) return false; I agree with sending events even if isEventProcessingActive() is false, but I would prefer to do this change in a different commit. It is cleaner if an Iceberg related commit has no non-Iceberg related side-effects. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 15:23:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8880/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 10:27:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 5: Sending INSERT events when Impala's event processor is not active revelaled a bug in the existing code. This is also fixed by the new PS. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 10 Jun 2021 10:11:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Hello Qifan Chen, Vihang Karajgaonkar, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17560 to look at the new patch set (#5). Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. IMPALA-10735: Fix Iceberg insert when event polling is enabled INSERT INTO Iceberg table is broken when HMS event polling is enabled. This patch disables INSERT event generation for Iceberg tables because Hive replication doesn't support Iceberg tables anyway. This commit also enables sending INSERT events when Impala's event processor is not active. This revealed a bug, i.e. INSERTs with empty result sets caused IllegalStateException. E.g.: create table ctas_empty as select * from functional.alltypes limit 0; This patch also fixes the above bug by turning a Precondition check to an 'if' stmt. Testing: * added e2e test Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 31 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/17560/5 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7212/ -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 21:38:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7212/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 17:00:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 17:00:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 3: Thanks, Everyone! -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 16:59:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 3: Code-Review+2 +1 Bumping up to +2 using Qifan's vote. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 16:51:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5023 PS1, Line 5023: > We only generate insert events for FeFsTables. Ah I see. Thanks for the clarification. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 16:50:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/17560/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5021 PS2, Line 5021: if (table instanceof FeIcebergTable) return false; : if (!BackendConfig.INSTANCE.isInsertEventsEnabled()) return false; : return true; > Vihang asked me to remove catalog_.isEventProcessingActive(). Done. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 14:33:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8869/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 14:30:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17560/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5021 PS2, Line 5021: if (table instanceof FeIcebergTable) return false; : if (!BackendConfig.INSTANCE.isInsertEventsEnabled()) return false; : return true; > Seems to me the test on iceberg tables can be done first, regardless of the Vihang asked me to remove catalog_.isEventProcessingActive(). I switched the order. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 14:10:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Hello Qifan Chen, Vihang Karajgaonkar, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17560 to look at the new patch set (#3). Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. IMPALA-10735: Fix Iceberg insert when event polling is enabled INSERT INTO Iceberg table is broken when HMS event polling is enabled. This patch disables INSERT event generation for Iceberg tables because Hive replication doesn't support Iceberg tables anyway. Testing: * added e2e test Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 25 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/17560/3 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17560/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5021 PS2, Line 5021: if (!BackendConfig.INSTANCE.isInsertEventsEnabled()) return false; : if (table instanceof FeIcebergTable) return false; : return true; Seems to me the test on iceberg tables can be done first, regardless of the current configuration/state of event processing. This is because the feature is not support for any iceberg tables. Also wonder why catalog_.isEventProcessingActive() is dropped. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 12:58:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8866/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 10:08:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 1: (2 comments) Thanks for the comments! http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5021 PS1, Line 5021: if (!catalog_.isEventProcessingActive()) return false; > Since you are modified this code anyways, I was thinking we should remove t Done http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5023 PS1, Line 5023: FeIcebergTable > I wonder why do we not hit this exception for Hbase or KuduTable. We only generate insert events for FeFsTables. Unlike IcebergTable, HBaseTable and KuduTable don't extend FeFsTable. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 09 Jun 2021 09:47:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Hello Vihang Karajgaonkar, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17560 to look at the new patch set (#2). Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. IMPALA-10735: Fix Iceberg insert when event polling is enabled INSERT INTO Iceberg table is broken when HMS event polling is enabled. This patch disables INSERT event generation for Iceberg tables because Hive replication doesn't support Iceberg tables anyway. Testing: * added e2e test Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 25 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/17560/2 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5021 PS1, Line 5021: if (!catalog_.isEventProcessingActive()) return false; Since you are modified this code anyways, I was thinking we should remove this. Generating insert events should not be conditional to events processor being active or not. http://gerrit.cloudera.org:8080/#/c/17560/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5023 PS1, Line 5023: FeIcebergTable I wonder why do we not hit this exception for Hbase or KuduTable. It probably makes sense if we change this condition as: if (table instanceof HdfsTable) return true; return false; -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 08 Jun 2021 18:26:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17560 ) Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 08 Jun 2021 16:42:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10735: Fix Iceberg insert when event polling is enabled
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17560 Change subject: IMPALA-10735: Fix Iceberg insert when event polling is enabled .. IMPALA-10735: Fix Iceberg insert when event polling is enabled INSERT INTO Iceberg table is broken when HMS event polling is enabled. This patch disables INSERT event generation for Iceberg tables because Hive replication doesn't support Iceberg tables anyway. Testing: * added e2e test Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/custom_cluster/test_event_processing.py 2 files changed, 26 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/17560/1 -- To view, visit http://gerrit.cloudera.org:8080/17560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icf4c3d7d4c0edf0d45628cf6d44785bad1049d76 Gerrit-Change-Number: 17560 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
