yangshangqing95 commented on PR #17282: URL: https://github.com/apache/iceberg/pull/17282#issuecomment-5122445905
> `puffin` is the file format, i think it might be better to make the name more generic. maybe something like `statistics_files` and `all_statistics_files` https://iceberg.apache.org/spec/#table-statistics Hi @kevinjqliu Thanks, that is a good point. I agree that statistics_files would align well with the terminology in the Iceberg spec if the table only exposed files registered through Table#statisticsFiles(). However, the current scope also includes Puffin files containing deletion vectors referenced by delete manifests, which are not statistics files. More importantly, Puffin is designed as an extensible container format, and it is likely that additional standard blob types and metadata use cases will be introduced over time. Naming the table statistics_files would couple its identity to one current use of Puffin and could become increasingly misleading as new blob types are added. The intent of puffin_files is instead to provide file-level observability for Puffin containers associated with a snapshot, independent of the logical content stored in them. The source and referenced blob type columns distinguish how each file is used. This also allows future Puffin-backed metadata to be exposed through the same table without introducing a new metadata table for every blob type or renaming the table as its scope evolves. If the preferred model is to expose only logical table-statistics registrations, then I agree that a narrower statistics_files table would be appropriate. But for a unified view covering statistics, deletion vectors, and future Puffin-backed metadata, I think puffin_files is the more accurate and extensible name. Similarly, I also implemented Puffin files. https://github.com/apache/iceberg/pull/17192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
