emkornfield commented on code in PR #588: URL: https://github.com/apache/parquet-format/pull/588#discussion_r3560032745
########## ParxMagicNumber.md: ########## @@ -0,0 +1,89 @@ +# PARX Parquet Format Specification + +This specification details a new magic number and associated fixed length footer metadata changes +that accompany the footer. + +## Motivation + +Most parts of the parquet specification lend themselves naturally to compatibility checks +when a new feature is added (e.g. encodings and compression values have an enum value added) +and fail appropriately. +However, some semantic changes or footer changes are impossible to communicate appropriately +within existing structures (e.g. changing the serialization of the footer). The motivation +for the new magic number and layout is to accomodate the latter set of changes by introducing +a new extensible mechanism for readers to detect these changes and fail accordingly. + +## Design Motivations + +* Provide a mechanism to only introduce a single new magic number for parquet that can + last at least a decade. +* Provide integrity checks for the footer. +* Provide the ability for readers to have a granular understanding of structural and semantic + backward incompatible features that are required to read a particular file. + +## File Layout + +A PARX file has the same overall structure as a standard Parquet file, with two differences: +the leading and trailing magic bytes are `PARX` instead of `PAR1/PARE`, and the trailing footer is +16 bytes instead of 8. + +The file layout is as follows: + +``` ++-----------+-------------------+--------------------+-------------+ +| 'PARX' | File Data | Footer Metadata | Footer tail | +| (4 bytes) | (variable length) | (variable length) | (16 bytes) | ++-----------+----------+--------+--------------------+-------------+ +``` + +All multi-byte integer fields are **little-endian**. + + +### PARX Footer Tail — 16 bytes + +``` ++------------------+-----------+----------+--------+ +| metadata_len | flags | crc32 | 'PARX' | ++------------------+-----------+----------+--------+ + offset 0 offset 4 offset 8 offset 12 Review Comment: One thing that was called out is that default Thrift parsers will fail hard if they encounter a new unknown enum for a required field. We should probably add a version number here, so that users can provide a useful error message e.g. something like: ``` Failed to parse footer. New version XX detected. ``` but I think the order of operations should still be: 1. If I support all falgs, try to parse the footer. 2. If footer parsing fails check version for appropriate error message. This allows readers with a custom thrift parsers to still potentially read most of the file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
