sfc-gh-sgrafberger commented on code in PR #585: URL: https://github.com/apache/parquet-format/pull/585#discussion_r3460832583
########## LogicalTypes.md: ########## @@ -635,7 +635,76 @@ The type has two type parameters: The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max statistics should be saved for this type and if such non-compliant statistics -are found during reading, they must be ignored. +are found during reading, they must be ignored. + +### FILE + +`FILE` annotates a group that represents a reference to an external file, along with +the minimum metadata required to read it. It is intended for use cases such as +storing file inventories, manifests, and unstructured data references (e.g., images +or audio files stored in object storage). + +The annotated group must contain the following fields, identified by name. Field IDs +may also be used for projection: + +| Field | Type | Required | +|----------|--------|----------| +| `path` | STRING | Yes | +| `size` | INT64 | No | +| `offset` | INT64 | No | +| `etag` | STRING | No | + Review Comment: One argument from the Parquet sync last week against this concern was that most engines anyway just infer content_type based on the file extension. However, this spec PR includes an `offset` field and says a reader should "seek to this offset and read size bytes to retrieve the referenced data" within the referenced file. But this means that a single physical container file can pack byte ranges for multiple logical files, and path then points to the container, not the logical entry. In that case the extension of path tells you nothing about the modality of a given slice. The container can hold images, audio, and documents at different offsets. Extension-based inference only works when one path = one homogeneous file, which offset/size explicitly break. That leaves either byte-sniffing each slice at query time, or carrying the type in side metadata. The second option seems to be something we want to avoid: a reader can read the FILE struct but still can't tell how to interpret the bytes at offset/size without additional knowledge, which breaks the "if you can read the type, you can understand everything in the type" goal. So it seems to me that offset/size make content_type necessary, at least whenever offset is non-zero, for the type to stay self-describing. Is there something I'm missing that keeps the bytes interpretable without it? Or is the officially recommended path to always do byte sniffing at query runtime? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
