[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 22:47:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Reviewed-on: http://gerrit.cloudera.org:8080/19380 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 98 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 9 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8962/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 17:37:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 8 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 17:37:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 12 Jan 2023 16:55:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12149/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 11 Jan 2023 23:56:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 7: The gerrit-verify-dryrun tests run the dockerised tests which use local catalog mode. This exposed a bug where my time travel changes were broken in local catalog mode. Fix this in LocalIcebergTable by calling the existing transfromToTHdfsTable() method. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 11 Jan 2023 23:43:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#7). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 98 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/7 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Verify tests fail in test_time_travel. It seems like this test is being run with local catalog mode. I can reproduce by running with local catalog mode. I will investigate further... -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 11 Jan 2023 04:04:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8958/ -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 23:05:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/12139/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 18:09:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Code-Review+2 Carry forward +2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 17:49:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#6). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 98 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/6 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8958/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 17:53:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 5: Code-Review+2 LGTM and nice tests! -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 14:06:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 5: Code-Review+1 Thanks for the work, LGTM! -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 10 Jan 2023 08:12:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12129/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 09 Jan 2023 22:48:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 16 files changed, 1,036 insertions(+), 99 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/5 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/19380 )
Change subject: IMPALA-10893: Use old schema during iceberg time travel.
..
Patch Set 4:
(3 comments)
Thanks for the comments
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309
PS2, Line 1309: timeTravelSpec.analyze(this);
> Splitting out to a separate function is nice, but I would call it in the co
OK I will inline the readSchema() call, thanks!
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@453
PS4, Line 453: Path resolvedPath =
analyzer_.resolvePathWithMasking(slotRef.getRawPath(),
: PathType.SLOT_REF, null);
> We could add a similar overload for these guys too.
Done
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java:
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java@197
PS4, Line 197: class ForwardingFeIcebergTable implements FeIcebergTable {
: private final FeIcebergTable base;
> I agree 100% with the pattern, I just wish there was a better way to do it.
It is a lot of boilerplate, but at least the boilerplate code is in one place
so is easy to maintain.
--
To view, visit http://gerrit.cloudera.org:8080/19380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
Gerrit-Change-Number: 19380
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 09 Jan 2023 22:29:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 4: Code-Review+1 Nice work, LGTM, added a few nitpicks -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 09 Jan 2023 08:46:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Gergely Fürnstáhl has posted comments on this change. (
http://gerrit.cloudera.org:8080/19380 )
Change subject: IMPALA-10893: Use old schema during iceberg time travel.
..
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309
PS2, Line 1309: timeTravelTable.readSchema();
> I did that initially, but then I split it out.
Splitting out to a separate function is nice, but I would call it in the
constructor.
Deferred construction puts in extra layer of complexity to the usage of the
object (we can forget to call this in the future).
In my opinion, it is worth it if
- the deferred part is computation-heavy
- and the object is "sane" and usable without it in some cases.
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@453
PS4, Line 453: Path resolvedPath =
analyzer_.resolvePathWithMasking(slotRef.getRawPath(),
: PathType.SLOT_REF, null);
We could add a similar overload for these guys too.
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java:
http://gerrit.cloudera.org:8080/#/c/19380/4/fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java@197
PS4, Line 197: class ForwardingFeIcebergTable implements FeIcebergTable {
: private final FeIcebergTable base;
I agree 100% with the pattern, I just wish there was a better way to do it. C++
private inheritance comes close, but still need to "reimplement"/change/extend
the API with forwarding to the base class.
--
To view, visit http://gerrit.cloudera.org:8080/19380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
Gerrit-Change-Number: 19380
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 09 Jan 2023 08:46:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12122/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:38:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 2: (5 comments) Thanks for review comments. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java File fe/src/main/java/org/apache/impala/analysis/Analyzer.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309 PS2, Line 1309: timeTravelTable.readSchema(); > Could it be part of the constructor of IcebergTimeTravelTable? I did that initially, but then I split it out. I think I prefer having the constructor be simple, and having work done in a method. If you feel strongly I will include it, but otherwise I'll leave this. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@95 PS2, Line 95: , null) > Maybe we could overload Analyzer.resolvePath() so we wouldn't need to chang Yes, that's nicer, done, thanks. http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java File fe/src/main/java/org/apache/impala/analysis/TableRef.java: http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java@471 PS2, Line 471: so we : // cannot check it yet. > "so we cannot check if the table supports time travel"? Done http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py@1844 PS2, Line 1844: > nit: +2 indent Done http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py File tests/query_test/test_iceberg.py: http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py@456 PS2, Line 456: # FIXME use new ts?? > Please resolve this. Does the following query work? Thanks, done, it worked. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:37:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12121/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Jan 2023 22:22:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#4). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 18 files changed, 1,035 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/4 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Tamas Mate, Gabor Kaszab, Zoltan Borok-Nagy, Gergely Fürnstáhl, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#3). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 18 files changed, 1,035 insertions(+), 104 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/3 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19380 )
Change subject: IMPALA-10893: Use old schema during iceberg time travel.
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19380/3/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19380/3/tests/query_test/test_iceberg.py@458
PS3, Line 458:
flake8: W291 trailing whitespace
http://gerrit.cloudera.org:8080/#/c/19380/3/tests/query_test/test_iceberg.py@458
PS3, Line 458: SELECT *, NULL FROM {tbl} FOR SYSTEM_TIME
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/19380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
Gerrit-Change-Number: 19380
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 06 Jan 2023 22:03:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/19380 )
Change subject: IMPALA-10893: Use old schema during iceberg time travel.
..
Patch Set 2:
(5 comments)
Left a few nit comments, otherwise the change looks great!
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@1309
PS2, Line 1309: timeTravelTable.readSchema();
Could it be part of the constructor of IcebergTimeTravelTable?
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@95
PS2, Line 95: , null)
Maybe we could overload Analyzer.resolvePath() so we wouldn't need to change
unrelated code paths.
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/19380/2/fe/src/main/java/org/apache/impala/analysis/TableRef.java@471
PS2, Line 471: so we
: // cannot check it yet.
"so we cannot check if the table supports time travel"?
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/authorization/test_ranger.py@1844
PS2, Line 1844:
nit: +2 indent
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19380/2/tests/query_test/test_iceberg.py@456
PS2, Line 456: # FIXME use new ts??
Please resolve this. Does the following query work?
SELECT * FROM {tbl} FOR SYSTEM_TIME AS OF
MINUS
SELECT *, NULL FROM {tbl} FOR SYSTEM_TIME AS OF
Or since the second subquery has NULL the whole query returns every row? Then
maybe we could
SELECT i FROM {tbl} FOR SYSTEM_TIME AS OF
MINUS
SELECT i FROM {tbl} FOR SYSTEM_TIME AS OF
--
To view, visit http://gerrit.cloudera.org:8080/19380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab
Gerrit-Change-Number: 19380
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 05 Jan 2023 16:57:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12048/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Dec 2022 20:21:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19380 to look at the new patch set (#2). Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 21 files changed, 1,030 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/2 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/12046/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Dec 2022 19:07:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19380 Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. IMPALA-10893: Use old schema during iceberg time travel. Before this change the schema used during Iceberg Time Travel was the current schema of the table. With this change we will use the schema from the point specified by the Time Travel parameters. The parameters used by an Iceberg Time Travel query are part of the FROM clause of the query. Previously analysis of the Time Travel parameters took place after the table Path was resolved, at which point some schema information is cached. In order to use the old schema during iceberg time travel however we need to ensure that the version of the Table that is used is always the version specified by the Time Travel parameters. To do this we have to move the analysis of the Time Travel parameters inside the code that resolves the Path. Add a new implementation of FeIcebergTable that represents an Iceberg table involved in Time Travel. This is implemented by embedding a reference to the base Iceberg Table. All methods that are not Time Travel related are delegated to the base table. The Time Travel related methods use the historic Iceberg schema. TESTING: - Add a new file iceberg_util.py to hold the snapshot utility code that was developed for the in-progress IMPALA-11482. - Extend the existing Iceberg Time Travel tests to check the schema. - Add a test that shows time travel working with columns masking. The column masking configuration is not tightly coupled to the schema so it is possible to mask historical columns. Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/analysis/UnnestExpr.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergPositionDeleteTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M tests/authorization/test_ranger.py M tests/common/iceberg_test_suite.py M tests/query_test/test_iceberg.py A tests/util/iceberg_util.py 21 files changed, 1,030 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/19380/1 -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman
[Impala-ASF-CR] IMPALA-10893: Use old schema during iceberg time travel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19380 ) Change subject: IMPALA-10893: Use old schema during iceberg time travel. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/19380/1/tests/util/iceberg_util.py File tests/util/iceberg_util.py: http://gerrit.cloudera.org:8080/#/c/19380/1/tests/util/iceberg_util.py@28 PS1, Line 28: 2020-08-30 22:58:08.44000 8270633197658268308 NULL TRUE tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/19380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7cbef6e20bbb567e517744fb1f34d880970399ab Gerrit-Change-Number: 19380 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Dec 2022 18:48:09 + Gerrit-HasComments: Yes
