tomerr90 commented on issue #15900: URL: https://github.com/apache/iceberg/issues/15900#issuecomment-4292572307
Thanks Sung! I feel like I have to be missing something here, an Iceberg table partitioned by a function over a timestamp field is probably the most common use case of Iceberg, so I find it very weird that this common case basically doesnt work in Flink upsert because it forces you to use the timestamp field (without the function even, just the field) as an equality field for the delete. @mxm @rodmeneses @Guosmilesmile sorry for the tag, PLMK if thats not OK, I saw you guys worked in the IcebergSink recently, would greatly appreciate your take on the above, seems like this "double" use of equalityFields for deletion equality and also flink stream partitioning is limiting common use cases, maybe the `keyBy` in the sink should transparently add the partitioning functions of the table AND the equalityFields? So equality is still only checked by the equalityFields but we maintain correctness of the deletes. Also, in general, this whole discussion is only relevant if writeParallelism is > 1, agree? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
