fallintoplace opened a new pull request, #3706: URL: https://github.com/apache/iceberg-python/pull/3706
# Rationale for this change `InspectTable.partitions()` aggregates files that belong to the same partition and records the snapshot that most recently updated that partition. The update condition compared `last_updated_snapshot_id` with the next snapshot's `timestamp_ms`. Snapshot IDs and commit timestamps are unrelated values, so the result depended on manifest-entry order and could retain stale snapshot metadata. The Apache Iceberg Java implementation orders these updates by the snapshot commit timestamp and updates the partition spec ID together with the timestamp and snapshot ID. This change applies the same behavior in PyIceberg: compare `last_updated_at` with `snapshot.timestamp_ms`, then update `spec_id`, `last_updated_at`, and `last_updated_snapshot_id` as one unit. ## Are these changes tested? Yes. A regression test processes an older snapshot before a newer snapshot while deliberately using decreasing snapshot IDs. It verifies that the newer commit timestamp determines all three snapshot-related fields. The following checks pass locally: - `PYTHONPATH=. uv run pytest tests/table/test_inspect.py -q` (5 passed) - `PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k "inspect_partition" -q` (2 passed) - `PYTHONPATH=. uv run --extra datafusion --extra pyiceberg-core pytest tests/table -q` (328 passed) - `make lint` ## Are there any user-facing changes? Yes. `table.inspect.partitions()` now reports the latest `spec_id`, `last_updated_at`, and `last_updated_snapshot_id` for each partition regardless of manifest-entry order. There are no API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
