danielcweeks commented on code in PR #603: URL: https://github.com/apache/parquet-format/pull/603#discussion_r3729853446
########## LogicalTypes.md: ########## @@ -756,10 +756,44 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +A schema that permits self-references must include the `inline` field. + +Each self-reference inherits the compression and encryption settings of the `inline` +column chunk in the same row group. The corresponding position is the position +representing the same `FILE` value in the `inline` column's repetition and definition +level stream. + +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. Review Comment: I don't think this will likely ever be needed. It's trivial to workaround the framing issues and working with File should indicate that there is an expectation that large values may be present. Readers need to be very permissive if they're going to support this type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
