[ https://issues.apache.org/jira/browse/IMPALA-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788948#comment-17788948 ]
ASF subversion and git services commented on IMPALA-12557: ---------------------------------------------------------- Commit a9ad48484a739cc66caf573cc30afa298cd0b767 in impala's branch refs/heads/master from Zoltan Borok-Nagy [ https://gitbox.apache.org/repos/asf?p=impala.git;h=a9ad48484 ] IMPALA-12557: DELETE throws DateTimeParseException when deleting from time-partitioned table There's a bug in IcebergDeleteSink that prevents Impala from successfully executing a DELETE operation on Iceberg tables. During DELETE we retrieve partition values from the virtual column ICEBERG__PARTITION__SERIALIZED. This contains the transformed values, e.g. in case of DAY-partitioning it contains the number of days since the UNIX epoch. Currently IcebergDeleteSink just uses these values as they are. There are two problems with this. First, we want to place the delete files under human-readable partition directories like other engines do, and like our own INSERT statement does. I.e. we want a partition directory /ts_day=2023-11-11/ and not /ts_day=19672/. The other problem is that 'IcebergUtil.partitionDataFromDataFile()' also expects the human-readable representation. This could be resolved at the CatalogD side to just accept the integer values, but then we would still need some logic in the IcebergDeleteSink to generate the human-readable values for the file paths. Moreover, partition values from INSERT statements are also received in the human-readable representation at the Catalog. This patch fixes the error by adding functions that transforms the partition values to their human-readable representations. This is done in the IcebergDeleteSink, so the Catalog-side logic is not affected. The above only affects the time-based transforms (YEAR, MONTH, DAY, HOUR), as other partition transform values don't use different representations. Some notes on HOUR transform and daylight saving time: There is no 1:1 mapping between an offset and the human-readable representation in a timezone that has daylight saving time. This is not an issue, as Impala's TIMESTAMP type is timezone-less. This also won't be an issue for the TIMESTAMPTZ type as timestamp values are normalized to UTC when stored, and UTC doesn't have daylight saving time. Testing: * C++ backend tests * E2E tests for all time-based transforms, and also partition evolution * Also added an extra test about TRUNCATEing numeric values which was untested Change-Id: I1cfeaed6409289663eb0f65b1ee2ecebd93e6118 Reviewed-on: http://gerrit.cloudera.org:8080/20711 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > DELETE throws DateTimeParseException when deleting from DAY-partitioned > Iceberg table > ------------------------------------------------------------------------------------- > > Key: IMPALA-12557 > URL: https://issues.apache.org/jira/browse/IMPALA-12557 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Zoltán Borók-Nagy > Assignee: Zoltán Borók-Nagy > Priority: Major > Labels: impala-iceberg > > DELETE throws DateTimeParseException when deleting from DAY-partitioned > Iceberg table. > Stack trace: > {noformat} > [1] java.time.format.DateTimeFormatter.parseResolved0 > (DateTimeFormatter.java:1,949) > [2] java.time.format.DateTimeFormatter.parse (DateTimeFormatter.java:1,851) > [3] java.time.LocalDate.parse (LocalDate.java:400) > [4] org.apache.iceberg.expressions.Literals$StringLiteral.to > (Literals.java:495) > [5] org.apache.iceberg.types.Conversions.fromPartitionString > (Conversions.java:70) > [6] org.apache.impala.util.IcebergUtil.getPartitionValue > (IcebergUtil.java:748) > [7] org.apache.impala.util.IcebergUtil.partitionDataFromDataFile > (IcebergUtil.java:726) > [8] org.apache.impala.service.IcebergCatalogOpExecutor.deleteRows > (IcebergCatalogOpExecutor.java:364) > [9] org.apache.impala.service.IcebergCatalogOpExecutor.execute > (IcebergCatalogOpExecutor.java:337) > [10] org.apache.impala.service.CatalogOpExecutor.updateCatalogImpl > (CatalogOpExecutor.java:7,042) > [11] org.apache.impala.service.CatalogOpExecutor.updateCatalog > (CatalogOpExecutor.java:6,786) > [12] org.apache.impala.service.JniCatalog.lambda$updateCatalog$15 > (JniCatalog.java:487) > ...{noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org