yangshangqing95 opened a new issue, #17339: URL: https://github.com/apache/iceberg/issues/17339
### Feature Request / Improvement ## Motivation Iceberg stores several types of metadata in Puffin files, including table statistics and deletion vectors. These files are referenced through different parts of Iceberg metadata: * Statistics files are registered in table metadata. * Deletion-vector Puffin files are referenced by live delete-file entries in delete manifests. There is currently no metadata table that provides a unified view of the Puffin file references associated with a table snapshot. Users must inspect table metadata JSON, manifest lists, and delete manifests separately to answer questions such as: * Which Puffin files are referenced by the current or a historical snapshot? * Which references come from registered statistics and which come from deletion vectors? * Which snapshots reference the same physical Puffin file? * How many distinct Puffin blobs in a file are referenced by a snapshot? * Which blob types and table fields are associated with those references? ## Proposal Add two metadata tables: ### `puffin_files` Exposes Puffin file references associated with one selected snapshot. The selected snapshot is the table's current snapshot by default and may be changed using the standard metadata-table time-travel APIs. ### `all_puffin_files` Exposes Puffin file references associated with all snapshots currently retained in table metadata. This follows the existing relationship between snapshot-scoped and all-snapshot metadata tables. ## Metadata sources The initial implementation supports two metadata sources: * `statistics`: statistics files currently registered in table metadata * `deletion_vector`: Puffin files referenced by live deletion-vector entries in delete manifests Additional Puffin-backed metadata sources may be added in the future without changing the basic table model. ## Row semantics Each row summarizes the Puffin blob references for one tuple of: * reference snapshot ID * metadata source * physical Puffin file path A physical Puffin file may therefore appear in multiple rows when: * it is referenced by multiple retained snapshots, or * it is referenced through multiple metadata sources. The blob count reports the number of distinct blobs referenced by that row. It is not necessarily the total number of blobs physically stored in the Puffin file. ## Proposed columns The two tables expose equivalent information, using `snapshot_id` for the snapshot-scoped table and `reference_snapshot_id` for the all-snapshots table: * `snapshot_id` / `reference_snapshot_id` * `file_path` * `source` * `file_size_in_bytes` * `referenced_blob_count` * `referenced_blob_types` * `referenced_fields` Each element of `referenced_fields` contains: * `field_id` * `current_field_name` Field IDs are authoritative. Field names are resolved against the current table schema and may be null when a referenced field has been dropped. ## Snapshot and history semantics Deletion-vector references are reconstructed independently for each retained snapshot from that snapshot's live delete manifests. Statistics files are obtained from the statistics files currently registered in table metadata and are associated with their registered snapshot IDs. Statistics registration may be changed independently of table snapshots. The metadata tables do not reconstruct historical statistics-registration state from previous metadata files. A statistics file that was replaced or removed is not reported merely because its associated data snapshot is still retained. If multiple branches or tags reference the same snapshot, `all_puffin_files` reports the Puffin references once for that snapshot ID rather than once per named reference. ## Planning The `all_puffin_files` table should support conservative task pruning for predicates over: * `reference_snapshot_id` * `source` A filter such as `source = 'statistics'` should avoid reading delete manifests. Snapshot predicates should avoid creating tasks for snapshots that cannot match. Unknown predicates must not cause incorrect pruning, including when they appear under `NOT`. ## Non-goals This proposal does not: * open Puffin files; * read Puffin footers; * parse or validate Puffin blob payloads; * expose blobs that are not referenced by Iceberg metadata; * reconstruct historical statistics-registration state; * verify whether referenced physical files currently exist; * identify files that are safe to delete; * modify statistics or deletion-vector metadata. All reported information is derived from Iceberg table metadata, snapshot manifest lists, and delete manifests. ## Implementation * [ ] #17192 — Add the snapshot-scoped `puffin_files` metadata table * [ ] #17282 — Add the all-snapshots `all_puffin_files` metadata table ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
