yangshangqing95 commented on PR #17282: URL: https://github.com/apache/iceberg/pull/17282#issuecomment-5122516401
> `puffin` is the file format, i think it might be better to make the name more generic. maybe something like `statistics_files` and `all_statistics_files` https://iceberg.apache.org/spec/#table-statistics Hi @kevinjqliu Thanks for the comment! I see the distinction you are making: ideally, a metadata-table name should describe a logical Iceberg concept rather than an implementation detail of the current physical format. A truly format-neutral alternative could be something like `auxiliary_files` and `all_auxiliary_files`, together with an explicit `file_format` column. That would leave room for other container formats in the future. My concern is that “auxiliary file” is not currently a defined Iceberg concept and its scope would be ambiguous. It could potentially include statistics files, partition statistics files, encryption-related files, index files, or other metadata artifacts. Introducing that abstraction would therefore require a broader definition of what qualifies as an auxiliary file, rather than being only a naming change. In contrast, Puffin already has a clear and established role in Iceberg as the container format for blobs referenced by table metadata and manifests. The rows exposed by these tables represent Puffin containers directly, and the schema includes container-specific information such as referenced blob count, blob types, and referenced field IDs. Puffin is also not limited to table statistics. It is already used for deletion vectors and is designed to support additional blob types in the future. For this reason, `puffin_files` and `all_puffin_files` describe the current row entity precisely. Given that Puffin is already a well-defined Iceberg concept, I am not sure we gain much by introducing a new, broader abstraction solely to avoid naming the file format. Would you still prefer defining a format-neutral auxiliary-file abstraction, including its intended scope, or does keeping the tables explicitly focused on Puffin files seem reasonable? and for `puffin_files` https://github.com/apache/iceberg/pull/17192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
