MatheusFreitas25 opened a new pull request, #3725:
URL: https://github.com/apache/iceberg-python/pull/3725
# Rationale for this change
`snapshot_by_id` does a linear scan over `self.snapshots`. It is called
once per
manifest entry by `InspectTable._process_manifest`, so
`inspect.partitions()` ends
up being O(data_files × snapshots) — the cost grows with the snapshot
count, which
means every commit makes all later reads slower.
On a table with ~317k data files and ~14.5k snapshots, reading the
partitions of a
single manifest went from 1.70s to 0.38s (~4.5x) with this change. A
lookup-only
benchmark of 20k calls shows where the time goes:
This replaces the scan with a memoized id-to-snapshot index.
A plain `cached_property` is not usable here: `model_copy` carries
`__dict__` over
to the new instance, so a copy that replaces `snapshots` would inherit a
stale
index and fail to find newly added snapshots. Since the commit path does
exactly
that (`snapshot_by_id` on the base metadata, then
`model_copy(update={"snapshots":
...})`), the index is tied to the list it was built from and recomputed
whenever
`snapshots` is a different list.
This follows the same pattern already used for `Schema._lazy_id_to_field`.
## Are these changes tested?
Three tests added to `tests/table/test_metadata.py`, using the existing
`example_table_metadata_v2` fixture
## Are there any user-facing changes?
No. `snapshot_by_id` returns the same `Snapshot` instance and `None` for
unknown
ids, exactly as before. No signature or API change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
