RussellSpitzer commented on code in PR #585: URL: https://github.com/apache/parquet-format/pull/585#discussion_r3483657310
########## LogicalTypes.md: ########## @@ -635,7 +635,76 @@ The type has two type parameters: The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max statistics should be saved for this type and if such non-compliant statistics -are found during reading, they must be ignored. +are found during reading, they must be ignored. + +### FILE + +`FILE` annotates a group that represents a reference to an external file, along with +the minimum metadata required to read it. It is intended for use cases such as +storing file inventories, manifests, and unstructured data references (e.g., images +or audio files stored in object storage). + +The annotated group must contain the following fields, identified by name. Field IDs +may also be used for projection: + +| Field | Type | Required | +|----------|--------|----------| +| `path` | STRING | Yes | +| `size` | INT64 | No | +| `offset` | INT64 | No | +| `etag` | STRING | No | + Review Comment: I think there's a strong case to include content_type here, but with well defined limits so it doesn't turn into a general optional-metadata bucket. Concretely: - content_type is a media type per [RFC 6838](https://www.rfc-editor.org/rfc/rfc6838) and should be an IANA-registered type (https://www.iana.org/assignments/media-types/media-types.xhtml). I'd lean toward constraining it further, but I don't think we can do that cleanly, and I'm not sure forbidding things like multipart actually buys anyone anything. - It is matched case-insensitively, readers should compare on the type/subtype. - If the field is missing, or a reader sees a value it can't parse or doesn't recognize, it is treated as application/octet-stream. That keeps readers from having to fail on a bad value, and it means "missing" and "octet-stream" get the same handling. Whatever an engine does for octet-stream is what it should do when the field is absent. The point of the constraint is that every engine, reader, or utility handed a FILE gets a known, bounded set of behaviors it can expect from the string, with a defined default. This is basically what the cloud storage providers do today, and what HTTP does with a missing Content-Type, so we're matching the most common existing behavior rather than inventing new semantics. To be clear on scope, this is not an argument for arbitrary optional metadata. This solves the self-describing type in the offset/size case Stefan raised, where path alone can't tell you the modality of a packed slice. cc @rdblue (who was chatting with me about this earlier today) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
