[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 22:47:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Reviewed-on: http://gerrit.cloudera.org:8080/19380 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 98 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 9 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8962/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 17:37:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 17:37:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 16:55:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12149/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 11 Jan 2023 23:56:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 7: The gerrit-verify-dryrun tests run the dockerised tests which use local catalog mode. This exposed a bug where my time travel changes were broken in local catalog mode. Fix this in LocalIcebergTable by calling the existing transfromToTHdfsTable() method. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 11 Jan 2023 23:43:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#7). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 98 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/7 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Verify tests fail in test_time_travel. It seems like this test is being run with local catalog mode. I can reproduce by running with local catalog mode. I will investigate further... -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 11 Jan 2023 04:04:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8958/ -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 23:05:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12139/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 18:09:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Code-Review+2 Carry forward +2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 17:49:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#6). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 98 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/6 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8958/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 17:53:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 5: Code-Review+2 LGTM and nice tests! -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 14:06:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 5: Code-Review+1 Thanks for the work, LGTM! -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 08:12:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12129/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 09 Jan 2023 22:48:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 99 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/5 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 4: (3 comments) Thanks for the comments http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309 PS2, Line 1309: timeTravelSpec.analyze(this); > Splitting out to a separate function is nice, but I would call it in the co OK I will inline the readSchema() call, thanks! http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@453 PS4, Line 453: Path resolvedPath = analyzer_.resolvePathWithMasking(slotRef.getRawPath(), : PathType.SLOT_REF, null); > We could add a similar overload for these guys too. Done http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java File fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java: http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java@197 PS4, Line 197: class ForwardingFeIcebergTable implements FeIcebergTable { : private final FeIcebergTable base; > I agree 100% with the pattern, I just wish there was a better way to do it. It is a lot of boilerplate, but at least the boilerplate code is in one place so is easy to maintain. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 09 Jan 2023 22:29:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 4: Code-Review+1 Nice work, LGTM, added a few nitpicks -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 09 Jan 2023 08:46:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309 PS2, Line 1309: timeTravelTable.readSchema(); > I did that initially, but then I split it out. Splitting out to a separate function is nice, but I would call it in the constructor. Deferred construction puts in extra layer of complexity to the usage of the object (we can forget to call this in the future). In my opinion, it is worth it if - the deferred part is computation-heavy - and the object is "sane" and usable without it in some cases. http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@453 PS4, Line 453: Path resolvedPath = analyzer_.resolvePathWithMasking(slotRef.getRawPath(), : PathType.SLOT_REF, null); We could add a similar overload for these guys too. http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java File fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java: http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java@197 PS4, Line 197: class ForwardingFeIcebergTable implements FeIcebergTable { : private final FeIcebergTable base; I agree 100% with the pattern, I just wish there was a better way to do it. C++ private inheritance comes close, but still need to "reimplement"/change/extend the API with forwarding to the base class. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 09 Jan 2023 08:46:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12122/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:38:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 2: (5 comments) Thanks for review comments. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309 PS2, Line 1309: timeTravelTable.readSchema(); > Could it be part of the constructor of IcebergTimeTravelTable? I did that initially, but then I split it out. I think I prefer having the constructor be simple, and having work done in a method. If you feel strongly I will include it, but otherwise I'll leave this. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@95 PS2, Line 95: , null) > Maybe we could overload Analyzer.resolvePath() so we wouldn't need to chang Yes, that's nicer, done, thanks. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java File fe/src/main/java/org/apache/impala/analysis/TableRef.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java@471 PS2, Line 471: so we : // cannot check it yet. > "so we cannot check if the table supports time travel"? Done http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py@1844 PS2, Line 1844: > nit: +2 indent Done http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py@456 PS2, Line 456: # FIXME use new ts?? > Please resolve this. Does the following query work? Thanks, done, it worked. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:37:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12121/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:22:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#4). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 18 files changed, 1,035 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/4 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#3). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 18 files changed, 1,035 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/3 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/19380/3/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/19380/3/tests/query_test/test_iceberg.py@458 PS3, Line 458: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/19380/3/tests/query_test/test_iceberg.py@458 PS3, Line 458: SELECT *, NULL FROM {tbl} FOR SYSTEM_TIME line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:03:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 2: (5 comments) Left a few nit comments, otherwise the change looks great! http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309 PS2, Line 1309: timeTravelTable.readSchema(); Could it be part of the constructor of IcebergTimeTravelTable? http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@95 PS2, Line 95: , null) Maybe we could overload Analyzer.resolvePath() so we wouldn't need to change unrelated code paths. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java File fe/src/main/java/org/apache/impala/analysis/TableRef.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java@471 PS2, Line 471: so we : // cannot check it yet. "so we cannot check if the table supports time travel"? http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py@1844 PS2, Line 1844: nit: +2 indent http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py@456 PS2, Line 456: # FIXME use new ts?? Please resolve this. Does the following query work? SELECT * FROM {tbl} FOR SYSTEM_TIME AS OF MINUS SELECT *, NULL FROM {tbl} FOR SYSTEM_TIME AS OF Or since the second subquery has NULL the whole query returns every row? Then maybe we could SELECT i FROM {tbl} FOR SYSTEM_TIME AS OF MINUS SELECT i FROM {tbl} FOR SYSTEM_TIME AS OF -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Jan 2023 16:57:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12048/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Dec 2022 20:21:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#2). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 21 files changed, 1,030 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12046/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Dec 2022 19:07:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19380 Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 21 files changed, 1,030 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/1 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19380/1/tests/util/iceberg_util.py File tests/util/iceberg_util.py: http://gerrit.cloudera.org:8080/#/c/19380/1/tests/util/iceberg_util.py@28 PS1, Line 28: 2020-08-30 22:58:08.44000 8270633197658268308 NULL TRUE tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Dec 2022 18:48:09 + Gerrit-HasComments: Yes