[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Reviewed-on: http://gerrit.cloudera.org:8080/16599 Reviewed-by: Zoltan Borok-Nagy Reviewed-by: wangsheng Tested-by: Impala Public Jenkins --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/data/README M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 14 files changed, 283 insertions(+), 15 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, approved wangsheng: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 18:18:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6695/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 14:03:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: Code-Review+2 Also LGTM -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 14:02:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: Code-Review+2 LGTM! -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 13:35:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7715/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 13:09:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6694/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 12:48:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 9: (1 comment) PS9 is a rebase with master to resolve merge conflict, http://gerrit.cloudera.org:8080/#/c/16599/7/testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json File testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json: http://gerrit.cloudera.org:8080/#/c/16599/7/testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json@4 PS7, Line 4: > The docekerised tests don't like the namenode hardcoded. We can easily remo Instead of adding this to the data load I created the table in the test and ran 2 inserts to have multiple snapshots. I found this way easier than to re-import the files for the table. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 12:46:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Qifan Chen, Zoltan Borok-Nagy, wangsheng, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#9). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/data/README M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 14 files changed, 283 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/9 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7714/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 11:35:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6693/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 23 Nov 2020 11:17:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/data/README M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 14 files changed, 283 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/8 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: I created this test table by Impala cherry-picking the write support patch from review. Since it got merget I'm wondering if it still makes sense to add this table to the dataload as I could create it easily in the tests with 2 consecutive inserts. I'm considering to remove this test table from the dataload and create it on the fly. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 15:29:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
..
Patch Set 7:
> (1 comment)
We need to use relative path when generate data files by Spark, here is a
simple demo, hope it helps:
val conf = new Configuration()
val warehouseLoc =
"/test-warehouse/iceberg_test/hadoop_catalog/hadoop_catalog_test"
val tblLoc =
"/test-warehouse/iceberg_test/hadoop_catalog/hadoop_catalog_test/functional_parquet/hadoop_catalog_test"
val catalog = new HadoopCatalog(conf, warehouseLoc);
val sparkSchema = StructType(List(StructField("id", IntegerType,true),
StructField("user", StringType,false),StructField("action", StringType,false),
StructField("event_time", TimestampType,false)))
val icebergSchema = SparkSchemaUtil.convert(sparkSchema)
val spec =
PartitionSpec.builderFor(icebergSchema).hour("event_time").identity("action").build
val table = catalog.createTable(TableIdentifier.of("functional_parquet",
"hadoop_catalog_test") ,icebergSchema, spec, null)
val action_data = Seq(
Row(1,"Alex","view",Timestamp.valueOf("2020-01-01 08:00:00"))
)
val df = spark.createDataFrame(sc.makeRDD(action_data), sparkSchema)
df.write.format("iceberg").option("write-format",
"parquet").mode("append").save(tblLoc)
spark.read.format("iceberg").load(tblLoc).show
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5
Gerrit-Change-Number: 16599
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Wed, 18 Nov 2020 12:28:27 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/16599/7/testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json File testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json: http://gerrit.cloudera.org:8080/#/c/16599/7/testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json@4 PS7, Line 4: hdfs://localhost:20500/ The docekerised tests don't like the namenode hardcoded. We can easily remove this from the JSON files, but to remove these from the manifest AVRO files we probably need to re-generate them with some different Iceberg configuration property. WangSheng was able to generate files this way, so he might know how to configure Iceberg. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 12:10:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6662/ -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 08:35:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6662/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 03:07:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 18 Nov 2020 03:06:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 17 Nov 2020 19:38:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7660/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 17 Nov 2020 08:43:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Qifan Chen, Zoltan Borok-Nagy, wangsheng, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#7). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/data/README A testdata/data/iceberg_test/iceberg_multi_snapshots/data/2b4c7b0f52b60a94-5308a4d8_52928493_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/data/97455b925ff67dcf-b56951ca_1396743290_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/559bbd83-47bf-420e-bd3b-0800ca89c81c-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/aa86e03c-db78-4ff2-afbd-905cbab0f4b7-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1034149846385246343-1-559bbd83-47bf-420e-bd3b-0800ca89c81c.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1860064961840897739-1-aa86e03c-db78-4ff2-afbd-905cbab0f4b7.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 26 files changed, 428 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/7 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 6: Code-Review+1 Thanks a lot for the rework! -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 16 Nov 2020 14:32:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7654/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 16 Nov 2020 13:37:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 6: (4 comments) http://gerrit.cloudera.org:8080/#/c/16599/5/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/16599/5/be/src/service/client-request-state.cc@440 PS5, Line 440: lt_row.__isset.colVals > Seems like this function doesn't take the TIMEZONE query option into accoun thanks for the investigation! It helped a lot. Done http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java: http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java@59 PS5, Line 59: specif > nit: 'specify' sounds better. Done http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1098 PS5, Line 1098: } > +1. Also the loadTable() call above. Good point! Done http://gerrit.cloudera.org:8080/#/c/16599/5/testdata/data/README File testdata/data/README: http://gerrit.cloudera.org:8080/#/c/16599/5/testdata/data/README@536 PS5, Line 536: > nit: Is it in contradiction to the line at 528: "impala cannot insert icebe Since IMPALA-10215 was submitted, we can write non-partitioned iceberg tables. Updated the comment in L528 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 16 Nov 2020 13:16:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Qifan Chen, Zoltan Borok-Nagy, wangsheng, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#6). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/data/README A testdata/data/iceberg_test/iceberg_multi_snapshots/data/2b4c7b0f52b60a94-5308a4d8_52928493_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/data/97455b925ff67dcf-b56951ca_1396743290_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/559bbd83-47bf-420e-bd3b-0800ca89c81c-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/aa86e03c-db78-4ff2-afbd-905cbab0f4b7-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1034149846385246343-1-559bbd83-47bf-420e-bd3b-0800ca89c81c.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1860064961840897739-1-aa86e03c-db78-4ff2-afbd-905cbab0f4b7.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 26 files changed, 428 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/6 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6648/ -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 14:04:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/16599/5/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/16599/5/be/src/service/client-request-state.cc@440 PS5, Line 440: ToStringFromUnixMicros Seems like this function doesn't take the TIMEZONE query option into account. I think you need to use TimestampValue::UtcFromUnixTimeMicros, then timestamp_value->UtcToLocal(timezone); http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1098 PS5, Line 1098: Set ancestorIds = Sets.newHashSet(SnapshotUtil.currentAncestors(table)) > It looks like this call can be placed before the start of the FOR loop. +1. Also the loadTable() call above. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 14:00:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6648/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 08:37:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has removed a vote on this change. Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Removed Code-Review+1 by wangsheng -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 08:11:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 12 Nov 2020 08:11:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
..
Patch Set 5: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16599/3/fe/src/main/jflex/sql-scanner.flex
File fe/src/main/jflex/sql-scanner.flex:
http://gerrit.cloudera.org:8080/#/c/16599/3/fe/src/main/jflex/sql-scanner.flex@152
PS3, Line 152: keywordMap.put("hudiparquet",
SqlParserSymbols.KW_HUDIPARQUET);
> I managed to address this. There was some conflict with the DESCRIBE (table
Thank you!
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5
Gerrit-Change-Number: 16599
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Tue, 10 Nov 2020 18:31:09 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: (3 comments) Looks good to me! http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java: http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java@59 PS5, Line 59: target nit: 'specify' sounds better. http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/16599/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1098 PS5, Line 1098: Set ancestorIds = Sets.newHashSet(SnapshotUtil.currentAncestors(table)) It looks like this call can be placed before the start of the FOR loop. http://gerrit.cloudera.org:8080/#/c/16599/5/testdata/data/README File testdata/data/README: http://gerrit.cloudera.org:8080/#/c/16599/5/testdata/data/README@536 PS5, Line 536: Created the table and inserted the data by Impala. nit: Is it in contradiction to the line at 528: "impala cannot insert iceberg table now"? -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 04 Nov 2020 20:37:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7608/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 03 Nov 2020 16:09:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16599/3/fe/src/main/jflex/sql-scanner.flex
File fe/src/main/jflex/sql-scanner.flex:
http://gerrit.cloudera.org:8080/#/c/16599/3/fe/src/main/jflex/sql-scanner.flex@152
PS3, Line 152: keywordMap.put("hudiparquet",
SqlParserSymbols.KW_HUDIPARQUET);
> We should be careful with adding new reserved words (it's best to avoid it)
I managed to address this. There was some conflict with the DESCRIBE (table)
statement and I had to merge them together and refactor a bit so that the
parser could accept everything as expected.
http://gerrit.cloudera.org:8080/#/c/16599/5/testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test
File
testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test:
http://gerrit.cloudera.org:8080/#/c/16599/5/testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test@17
PS5, Line 17: '2020-08-31 07:58:08.44'
Apparently, when I run these tests in jenkins this check fails because some
kind of time adjustment happens and "set timezone" above has no effect on the
result.
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5
Gerrit-Change-Number: 16599
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Tue, 03 Nov 2020 16:00:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Zoltan Borok-Nagy, wangsheng, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#5). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/data/README A testdata/data/iceberg_test/iceberg_multi_snapshots/data/2b4c7b0f52b60a94-5308a4d8_52928493_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/data/97455b925ff67dcf-b56951ca_1396743290_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/559bbd83-47bf-420e-bd3b-0800ca89c81c-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/aa86e03c-db78-4ff2-afbd-905cbab0f4b7-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1034149846385246343-1-559bbd83-47bf-420e-bd3b-0800ca89c81c.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1860064961840897739-1-aa86e03c-db78-4ff2-afbd-905cbab0f4b7.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 26 files changed, 421 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/5 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 4: I'm still working on not introducing a new keyword for "HISTORY". Currently there is some difficulties not to collide with DESCRIBE tablename in the .cup file generation. Let me see if I an find a workaround for this. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 03 Nov 2020 09:44:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 4: Code-Review+1 (1 comment) Thanks for changes, Gabor. Apart from ' adding new reserved words' problem which Tim mentioned above, this patch LGTM. http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG@14 PS1, Line 14: ++-+-+-+ : | creation_time | snapshot_id | parent_id | is_current_ancestor | : ++-+-+-+ : | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL | TRUE| : | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE| : ++-+-+-+ > That should be a different query in my opinion. Here what I try to reproduc Thanks for explain, maybe we can support snapshot in another patch. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Tue, 03 Nov 2020 02:22:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7601/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 02 Nov 2020 14:10:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/16599/2/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/16599/2/common/thrift/Frontend.thrift@292 PS2, Line 292: DESCRIBE HISTORY > DESCRIBE HISTORY Done http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java: http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java@31 PS2, Line 31: DESCRIBE HISTORY s > DESCRIBE HISTORY Done http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1077 PS2, Line 1077: DESCRIBE HISTORY q > DESCRIBE HISTORY Done -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 02 Nov 2020 13:58:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 4: (2 comments) PS4 is a rebase with master http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/client-request-state.cc@426 PS2, Line 426: DESCRIBE_HISTORY: > Should be DESCRIBE_HISTORY Done http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/frontend.h File be/src/service/frontend.h: http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/frontend.h@99 PS2, Line 99: Call > nit: typo Done -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 02 Nov 2020 13:54:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Zoltan Borok-Nagy, wangsheng, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#4). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex M testdata/data/README A testdata/data/iceberg_test/iceberg_multi_snapshots/data/2b4c7b0f52b60a94-5308a4d8_52928493_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/data/97455b925ff67dcf-b56951ca_1396743290_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/559bbd83-47bf-420e-bd3b-0800ca89c81c-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/aa86e03c-db78-4ff2-afbd-905cbab0f4b7-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1034149846385246343-1-559bbd83-47bf-420e-bd3b-0800ca89c81c.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1860064961840897739-1-aa86e03c-db78-4ff2-afbd-905cbab0f4b7.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 26 files changed, 403 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/4 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
..
Patch Set 3:
(1 comment)
THis is really cool! I had a concern about reserved words that hopefully isn't
too big a deal to address.
http://gerrit.cloudera.org:8080/#/c/16599/3/fe/src/main/jflex/sql-scanner.flex
File fe/src/main/jflex/sql-scanner.flex:
http://gerrit.cloudera.org:8080/#/c/16599/3/fe/src/main/jflex/sql-scanner.flex@152
PS3, Line 152: keywordMap.put("history", SqlParserSymbols.KW_HISTORY);
We should be careful with adding new reserved words (it's best to avoid it)
because it can cause problems if there are unquoted symbols in existing queries.
https://impala.apache.org/docs/build/html/topics/impala_reserved_words.html
I think you could avoid adding the keyword if you used an existing keyword
(change? commit? versioning?), or if you made it an IDENT token and compared it
to "history" on the Java side.
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5
Gerrit-Change-Number: 16599
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Fri, 30 Oct 2020 23:23:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7591/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 30 Oct 2020 14:26:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Zoltan Borok-Nagy, wangsheng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#3). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex M testdata/data/README A testdata/data/iceberg_test/iceberg_multi_snapshots/data/2b4c7b0f52b60a94-5308a4d8_52928493_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/data/97455b925ff67dcf-b56951ca_1396743290_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/559bbd83-47bf-420e-bd3b-0800ca89c81c-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/aa86e03c-db78-4ff2-afbd-905cbab0f4b7-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1034149846385246343-1-559bbd83-47bf-420e-bd3b-0800ca89c81c.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1860064961840897739-1-aa86e03c-db78-4ff2-afbd-905cbab0f4b7.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 26 files changed, 403 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/3 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7590/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 30 Oct 2020 13:51:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/client-request-state.cc@426 PS2, Line 426: SHOW_TABLE_HISTORY Should be DESCRIBE_HISTORY http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/frontend.h File be/src/service/frontend.h: http://gerrit.cloudera.org:8080/#/c/16599/2/be/src/service/frontend.h@99 PS2, Line 99: CALL nit: typo http://gerrit.cloudera.org:8080/#/c/16599/2/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/16599/2/common/thrift/Frontend.thrift@292 PS2, Line 292: SHOW CREATE TABLE DESCRIBE HISTORY http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java: http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java@31 PS2, Line 31: SHOW TABLE HISTORY DESCRIBE HISTORY http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/16599/2/fe/src/main/java/org/apache/impala/service/Frontend.java@1077 PS2, Line 1077: SHOW TABLE HISTORY DESCRIBE HISTORY -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 30 Oct 2020 13:44:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16599 ) Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. Patch Set 2: Changes the syntax from SHOW TABLE HISTORY to DESCRIBE HISTORY as that is what Delta on SparkSql uses. This was we have language compatibility at least one of the solutions on the field. -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 30 Oct 2020 13:35:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/16599 )
Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
..
Patch Set 2:
(12 comments)
Sorry, I think I have also posted a rebase along with my changes.
http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-10288
> I think maybe we should create a new sub-task under IMPALA-10153 for this p
Done
http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG@17
PS1, Line 17: 2020-10-13 14:01:07.234000
> Does Spark 'select * from table.history' show the same timestamp values? I
I do no conversion on the stored data. Whatever is stored (millis from epoch)
is taken as it is when I create a timestamp from it.
http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG@14
PS1, Line 14:
++-+-+-+
: | creation_time | snapshot_id | parent_id
| is_current_ancestor |
:
++-+-+-+
: | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL
| TRUE|
: | 2020-10-13 14:01:19.307000 | 4221472712544505868 |
4400379706200951771 | TRUE|
:
++-+-+-+
> Maybe we can add more info here, such as operation/manifest_list/summary. S
That should be a different query in my opinion. Here what I try to reproduce is
Spark's "select * from tablename.history" not "select * from
tablename.snapshots". However, if you still feel that those information should
be part of the output here, I'm open to include it.
See here https://iceberg.apache.org/spark/#inspecting-tables
I added this to the message.
http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG@24
PS1, Line 24: cting
> nit: basic
Done
http://gerrit.cloudera.org:8080/#/c/16599/1//COMMIT_MSG@24
PS1, Line 24:
:
> SHOW TABLE HISTORY?
Done
http://gerrit.cloudera.org:8080/#/c/16599/1/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/16599/1/be/src/service/client-request-state.cc@426
PS1, Line 426: case TCatalogOpType::SHOW_TABLE_HISTORY
> Maybe we should add some comments here, since this syntax is only supported
Done
http://gerrit.cloudera.org:8080/#/c/16599/1/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/16599/1/common/thrift/Frontend.thrift@287
PS1, Line 287: // Result of the DESCRIBE HISTORY command.
: struct TGetTableHistoryResult {
: 1: required list result
: }
:
> Please also add some comments for these new thrift structs.
Done
http://gerrit.cloudera.org:8080/#/c/16599/1/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/16599/1/fe/src/main/cup/sql-parser.cup@295
PS1, Line 295: KW_HISTORY
> Adding "HISTORY" as a keyword will fail table creation with field names of
Update: I just forgot to add KW_HISTORY to another place of this file.
http://gerrit.cloudera.org:8080/#/c/16599/1/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/16599/1/fe/src/main/java/org/apache/impala/service/Frontend.java@109
PS1, Line 109: import org.apache.impala.catalog.ImpaladCatalo
> Unused import
Done
http://gerrit.cloudera.org:8080/#/c/16599/1/fe/src/main/java/org/apache/impala/service/Frontend.java@1066
PS1, Line 1066: pendingCheckTasks.add(checkAuthorizationPool_.submit(
> Please also add some comments for this method.
Done
http://gerrit.cloudera.org:8080/#/c/16599/1/testdata/data/README
File testdata/data/README:
http://gerrit.cloudera.org:8080/#/c/16599/1/testdata/data/README@578
PS1, Line 578: full_acid_schema_but_no_acid_version.orc
> Please extend this file with the newly added Iceberg table.
Done
http://gerrit.cloudera.org:8080/#/c/16599/1/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/16599/1/testdata/datasets/functional/functional_schema_template.sql@2983
PS1, Line 2983: IES('iceberg.file_format'='orc', 'icebe
> Maybe we should use '/test-warehouse/iceberg_test/iceberg_multi_snapshots'
Done
--
To view, visit http://gerrit.cloudera.org:8080/16599
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e
[Impala-ASF-CR] IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables
Hello Zoltan Borok-Nagy, wangsheng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16599 to look at the new patch set (#2). Change subject: IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables .. IMPALA-10288: Implement DESCRIBE HISTORY for Iceberg tables The DESCRIBE HISTORY works for Iceberg tables and displays the snapshot history of the table. An example output: DESCRIBE HISTORY iceberg_multi_snapshots; ++-+-+-+ | creation_time | snapshot_id | parent_id | is_current_ancestor | ++-+-+-+ | 2020-10-13 14:01:07.234000 | 4400379706200951771 | NULL| TRUE | | 2020-10-13 14:01:19.307000 | 4221472712544505868 | 4400379706200951771 | TRUE | ++-+-+-+ The purpose here was to have similar output with this new feature as what SparkSql returns for "SELECT * from tablename.history". See "History" section of https://iceberg.apache.org/spark/#inspecting-tables Testing: - iceberg-negative.test was extended to check that DESCRIBE HISTORY is not applicable for non-Iceberg tables. - iceberg-table-history.test: Covers basic usage of DESCRIBE HISTORY. Tests on tables created with Impala and also with Spark. Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/DescribeHistoryStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex M testdata/data/README A testdata/data/iceberg_test/iceberg_multi_snapshots/data/2b4c7b0f52b60a94-5308a4d8_52928493_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/data/97455b925ff67dcf-b56951ca_1396743290_data.0.parq A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/559bbd83-47bf-420e-bd3b-0800ca89c81c-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/aa86e03c-db78-4ff2-afbd-905cbab0f4b7-m0.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1034149846385246343-1-559bbd83-47bf-420e-bd3b-0800ca89c81c.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/snap-1860064961840897739-1-aa86e03c-db78-4ff2-afbd-905cbab0f4b7.avro A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_multi_snapshots/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-table-history.test M tests/query_test/test_iceberg.py 26 files changed, 403 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/16599/2 -- To view, visit http://gerrit.cloudera.org:8080/16599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a4b92c27e8e4a79109696cbae62735a00750e5 Gerrit-Change-Number: 16599 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
