rdblue commented on code in PR #585:
URL: https://github.com/apache/parquet-format/pull/585#discussion_r3555463492
##########
src/main/thrift/parquet.thrift:
##########
@@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ * - path (STRING, required): an opaque string path to the file (e.g.
s3://bucket/file.jpg)
Review Comment:
@pitrou I think it's reasonable to clarify that we expect the location to be
a URI, but are you saying that implementations must parse and validate the
contents of the column? I don't think that we enforce things like that anywhere
else. But if I understand correctly, you're just saying that the field
definition should state that the value in the field is a URI and would not
require expensive validation. Is that right?
I don't want to set a precedent that logical types can require per-value
validation. That would mean a lot of work in the write path for little gain.
Validating a URI would require parsing it because the parts have different sets
of valid characters and escape requirements (`_` is not allowed in a scheme,
nor are escaped characters).
In the more general case of requiring per-value validation, we could
introduce negatives for forward compatibility. Avro lets you define enums that
are translated into ordinals when stored. This ended up being surprisingly
restrictive because adding a symbol breaks forward compatibility. In most
cases, readers using old schemas will automatically project and ignore new
columns. Using a string works without breaking the reader, but adding an enum
symbol causes a failure to resolve the read schema. Something similar could
easily happen if we require per-value validation: even widening that validation
is a forward-incompatible change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
