amitpoorab opened a new pull request, #3775:
URL: https://github.com/apache/iceberg-python/pull/3775
…partitioned tables
Regression: On a partitioned table with temporal transforms
(day/month/year/hour), upsert operations were leaving replaced rows in place
and duplicating untouched rows, silently. This was a regression from v0.11.1.
Root cause: Manifest pruning optimization (PR #3011) uses a predicate built
from partition VALUES aliased onto SOURCE COLUMN names, which causes false
negatives for non-identity transforms during manifest evaluation. For temporal
transforms, a day-ordinal (e.g. 20455) gets misinterpreted as
microseconds-since-epoch and produces a wildly different partition value when
re-transformed, causing the manifest evaluator to wrongly conclude 'this
manifest cannot match' and skip it, leaving stale rows in place.
Fix: Restrict the manifest pruning optimization to identity-transform-only
specs. For non-identity transforms, fall back to the pre-#3011 behavior of
always opening manifests and checking exact file identity, which is provably
correct. This mirrors an existing guard in dynamic_partition_overwrite.
Performance: identity-partitioned tables keep the #3011 speedup;
non-identity specs revert to 0.11.1 perf (slower but correct).
Added: parametrized regression test covering identity, day, month, year,
hour, and bucket transforms to prevent re-regression.
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${3758} below
with the actual Github issue id. -->
<!-- Closes #${3758} -->
# Rationale for this change
On partitioned tables with temporal transforms (day/month/year/hour),
upsert() was silently
corrupting data: leaving replaced rows in place and duplicating untouched
rows.
Root cause: PR #3011 added a manifest pruning optimization that builds
predicates by aliasing
partition VALUES onto SOURCE COLUMN names. This causes type mismatches for
non-identity
transforms during manifest evaluation, resulting in false negatives that
skip the exact
identity checks — leaving stale data behind.
Fix: Restrict the manifest pruning optimization to identity-transform-only
specs.
Non-identity specs fall back to the proven-correct pre-#3011 behavior
(slower but correct).
## Are these changes tested?
Yes. Added a parametrized regression test covering identity, day, month,
year, hour,
and bucket transforms. The test appends 2 rows sharing a partition, upserts
1 row,
and verifies no duplicates or stale rows remain.
All 28 upsert tests pass (22 existing + 6 new regression tests).
## Are there any user-facing changes?
Yes. This fixes a data corruption bug (regression from v0.11.1) in upsert()
on
temporal-partitioned tables. Users with such tables will now get correct
results instead
of silent data loss.
Performance: Identity-partitioned tables retain the #3011 optimization.
Non-identity
specs revert to v0.11.1 performance (slower but correct).
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
