alkis commented on PR #603: URL: https://github.com/apache/parquet-format/pull/603#issuecomment-5335551560
Change of direction: this PR now **removes** `FILE` self-references instead of specifying their storage semantics (d022d27). The AAD discussion is what convinced me. Making a self-referenced range a first-class encryption module meant the `FILE` type had to define its own compression block, its own encryption module, its own AAD identity, and its own size accounting — a parallel storage mechanism living next to pages but not being one. Every review comment here was a symptom of that: whether the range is bound to a page, what its identity is for AAD, whether the size is pre- or post-encryption, whether the decompressed size needs storing. Those questions only exist because the bytes sit outside any page while still needing page-like machinery. Out-of-band pages answer all of them once, in the page layer, for every type rather than just `FILE`. A large value gets stored in a page that lives outside the normal page sequence, and it inherits compression, encryption, AAD, and size accounting from the existing page rules with nothing new to specify. I will send that as a separate proposal. What this PR does now: - `offset` and `size` apply only to data referenced by `uri` - a byte range within the current file cannot be referenced - `Encryption.md` returns to its state on master - inline values and external references are unchanged @rok @wgtmac @danielcweeks @brkyvz — the open threads all point at text that no longer exists, so nothing there needs an answer from you. I have left them unresolved rather than resolving them myself, since the reasoning in them is the reason for this change and worth keeping visible. The parts of your feedback that survive independently of self-references are already in: the aligned resolution table (@brkyvz) and the normative `size` wording (@danielcweeks). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
