[ https://issues.apache.org/jira/browse/FLINK-35318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
linshangquan updated FLINK-35318: --------------------------------- Description: In our scenario, we have an Iceberg table that contains a column named 'time' of the {{timestamptz}} data type. This column has 10 rows of data where the 'time' value is {{'2024-04-30 07:00:00'}} expressed in the "Asia/Shanghai" timezone. !image-2024-05-09-14-06-58-007.png! We encountered a strange phenomenon when accessing the table using Iceberg-flink. When the {{WHERE}} clause includes the {{time}} column, the results are incorrect. ZoneId.{_}systemDefault{_}() = "Asia/Shanghai" !image-2024-05-09-14-09-38-453.png! When there is no {{WHERE}} clause, the results are correct. !image-2024-05-09-14-22-14-417.png! During debugging, we found that when a {{WHERE}} clause is present, a {{FilterPushDownSpec}} is generated, and this {{FilterPushDownSpec}} utilizes {{RexNodeToExpressionConverter}} for translation. !image-2024-05-09-14-11-38-476.png! When {{RexNodeToExpressionConverter#visitLiteral}} encounters a {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type, it uses the specified timezone "Asia/Shanghai" to convert the {{TimestampString}} type to an {{Instant}} type. However, the upstream {{TimestampString}} data has already been processed in UTC timezone. By applying the local timezone processing here, an error occurs due to the mismatch in timezones. Whether the handling of {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type of data in {{RexNodeToExpressionConverter#visitLiteral}} is a bug, and whether it should process the data in UTC timezone. Please help confirm if this is the issue, and if so, we can submit a patch to fix it. was: In our scenario, we have an Iceberg table that contains a column named 'time' of the {{timestamptz}} data type. This column has 10 rows of data where the 'time' value is {{'2024-04-30 07:00:00'}} expressed in the "Asia/Shanghai" timezone. !image-2024-05-09-14-06-58-007.png! We encountered a strange phenomenon when accessing the table using Iceberg-flink. When the {{WHERE}} clause includes the {{time}} column, the results are incorrect. ZoneId.{_}systemDefault{_}() = "Asia/Shanghai" !image-2024-05-09-14-09-38-453.png! When there is no {{WHERE}} clause, the results are correct. During debugging, we found that when a {{WHERE}} clause is present, a {{FilterPushDownSpec}} is generated, and this {{FilterPushDownSpec}} utilizes {{RexNodeToExpressionConverter}} for translation. !image-2024-05-09-14-11-38-476.png! When {{RexNodeToExpressionConverter#visitLiteral}} encounters a {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type, it uses the specified timezone "Asia/Shanghai" to convert the {{TimestampString}} type to an {{Instant}} type. However, the upstream {{TimestampString}} data has already been processed in UTC timezone. By applying the local timezone processing here, an error occurs due to the mismatch in timezones. Whether the handling of {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type of data in {{RexNodeToExpressionConverter#visitLiteral}} is a bug, and whether it should process the data in UTC timezone. Please help confirm if this is the issue, and if so, we can submit a patch to fix it. > incorrect timezone handling for TIMESTAMP_WITH_LOCAL_TIME_ZONE type during > predicate pushdown > --------------------------------------------------------------------------------------------- > > Key: FLINK-35318 > URL: https://issues.apache.org/jira/browse/FLINK-35318 > Project: Flink > Issue Type: Bug > Components: Table SQL / API > Affects Versions: 1.16.1 > Environment: flink version 1.16.1 > iceberg version 1.14.3 > Reporter: linshangquan > Priority: Major > Attachments: image-2024-05-09-14-06-58-007.png, > image-2024-05-09-14-09-38-453.png, image-2024-05-09-14-11-38-476.png, > image-2024-05-09-14-22-14-417.png > > > In our scenario, we have an Iceberg table that contains a column named 'time' > of the {{timestamptz}} data type. This column has 10 rows of data where the > 'time' value is {{'2024-04-30 07:00:00'}} expressed in the "Asia/Shanghai" > timezone. > !image-2024-05-09-14-06-58-007.png! > > We encountered a strange phenomenon when accessing the table using > Iceberg-flink. > When the {{WHERE}} clause includes the {{time}} column, the results are > incorrect. > ZoneId.{_}systemDefault{_}() = "Asia/Shanghai" > !image-2024-05-09-14-09-38-453.png! > When there is no {{WHERE}} clause, the results are correct. > !image-2024-05-09-14-22-14-417.png! > During debugging, we found that when a {{WHERE}} clause is present, a > {{FilterPushDownSpec}} is generated, and this {{FilterPushDownSpec}} utilizes > {{RexNodeToExpressionConverter}} for translation. > !image-2024-05-09-14-11-38-476.png! > When {{RexNodeToExpressionConverter#visitLiteral}} encounters a > {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type, it uses the specified timezone > "Asia/Shanghai" to convert the {{TimestampString}} type to an {{Instant}} > type. However, the upstream {{TimestampString}} data has already been > processed in UTC timezone. By applying the local timezone processing here, an > error occurs due to the mismatch in timezones. > Whether the handling of {{TIMESTAMP_WITH_LOCAL_TIME_ZONE}} type of data in > {{RexNodeToExpressionConverter#visitLiteral}} is a bug, and whether it should > process the data in UTC timezone. > > Please help confirm if this is the issue, and if so, we can submit a patch to > fix it. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)