Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18399 to look at the new patch set (#5). Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode ...................................................................... IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode IcebergScanNode interprets the timestamp literals as UTC timestamps during predicate pushdown to Iceberg. It causes problems when the Iceberg table uses TIMESTAMPTZ (which corresponds to TIMESTAMP WITH LOCAL TIME ZONE in SQL) because in the scanners we assume that the timestamp literals in a query are in local timezone. Hence, if the Iceberg table is partitioned by HOUR(ts), and Impala is running in a different timezone than UTC, then the following query doesn't return any rows: SELECT * from t WHERE ts = <some ts>; Because during predicate pushdown the timestamp is interpreted as a UTC timestamp (no conversion from local to UTC), but during query execution the timestamp data in the files are converted to local timezone, then compared to <some ts>. I.e. in the scanner the assumption is that <some ts> is in local timezone. On the other hand, when Iceberg type TIMESTAMP (which correcponds to TIMESTAMP WITHOUT TIME ZONE in SQL) is used, then we should just push down the timestamp values without any conversion. In this case there is no conversion in the scanners either. Testing: * added e2e test with TIMESTAMPTZ * added e2e test with TIMESTAMP Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa --- M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2021-01-10-12/00000-0-boroknagyz_20220413134343_7000310a-aecc-4e44-8c41-f8885675a9cb-job_16493406300920_0019-00002.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2021-10-31-02/00000-0-boroknagyz_20220413134343_7000310a-aecc-4e44-8c41-f8885675a9cb-job_16493406300920_0019-00001.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2022-04-11-00/00000-0-boroknagyz_20220413134343_7000310a-aecc-4e44-8c41-f8885675a9cb-job_16493406300920_0019-00003.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/data/ts_hour=2022-04-11-12/00000-0-boroknagyz_20220413134343_7000310a-aecc-4e44-8c41-f8885675a9cb-job_16493406300920_0019-00004.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/a2d413b6-0539-4ef9-8c70-df065ef63402-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/snap-7021219619084712613-1-a2d413b6-0539-4ef9-8c70-df065ef63402.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamp_part/metadata/version-hint.text A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2021-01-10-11/00000-0-boroknagyz_20220413131021_733ffe4f-ca07-441a-a714-b4bfe314ad19-job_16493406300920_0017-00003.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2021-10-31-00/00000-0-boroknagyz_20220413131021_733ffe4f-ca07-441a-a714-b4bfe314ad19-job_16493406300920_0017-00001.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2021-10-31-01/00000-0-boroknagyz_20220413131021_733ffe4f-ca07-441a-a714-b4bfe314ad19-job_16493406300920_0017-00002.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2022-04-10-22/00000-0-boroknagyz_20220413131021_733ffe4f-ca07-441a-a714-b4bfe314ad19-job_16493406300920_0017-00004.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/data/ts_hour=2022-04-11-10/00000-0-boroknagyz_20220413131021_733ffe4f-ca07-441a-a714-b4bfe314ad19-job_16493406300920_0017-00005.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/f522d668-4bd4-4ffa-b617-e1523753691a-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/snap-6467287209495292139-1-f522d668-4bd4-4ffa-b617-e1523753691a.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_timestamptz_part/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 25 files changed, 564 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18399/5 -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tma...@apache.org> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>