amitpoorab opened a new pull request, #3782:
URL: https://github.com/apache/iceberg-python/pull/3782
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #3758 -->
# Rationale for this change
On temporal-partitioned tables, `upsert()` was silently corrupting data by
leaving replaced rows in place and duplicating untouched rows. `Table.delete()`
on partial rewrites had the same issue.
**Root cause:** Manifest pruning derived partition predicates from source
columns + partition values, then re-applied transforms during projection. This
double-transformation caused false negatives in manifest evaluation, missing
the manifest holding the file being replaced.
**The fix:** Build partition-space predicates directly using partition field
names (e.g., `ts_day` instead of `ts`), avoiding re-transformation. This works
for ALL transform types (identity, temporal day/month/year/hour, bucket,
truncate) — not just identity.
**Credits:** Commits cherry-picked from @paulcaron16k's investigation
branch. They implemented the fix; I diagnosed the root cause and brought it
forward based on maintainer feedback.
## Are these changes tested?
Yes. Added comprehensive regression tests:
1. **`test_upsert_partial_rewrite_of_partitioned_file`** — Tests `upsert()`
on 7 transform types:
- IdentityTransform (control)
- TruncateTransform (idempotent, control)
- YearTransform, MonthTransform, DayTransform, HourTransform (temporal —
regression cases)
- BucketTransform (hash-based)
## Are there any user-facing changes?
Yes. This fixes a data corruption bug (regression from v0.11.1) in
`upsert()` and `delete()` operations on temporal-partitioned tables. Users with
day/month/year/hour-partitioned tables will now get correct results instead of
silent data loss.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
