+1 (non-binding) Best, Xuwei Fu
Alkis Evlogimenos via dev <[email protected]> 于2026年8月24日周一 22:03写道： > Hi folks, > > I would like to start a vote on adopting the format change in PR 603 [1]: > remove self-references from the FILE logical type. This came out of the > discussion on inheriting the inline codec [2]. > > FILE as merged lets a value set offset and size with no uri, which > addresses a byte range inside the containing file. This byte range is > effectively owned by the Parquet writer. Specifying that range properly > means giving it a compression block, an encryption module, an AAD identity, > and its own size accounting. That is a second page mechanism reachable only > through FILE. The use case is mid sized values and it is better served by > non-contiguous pages, which we will propose separately. Removing > self-references keeps FILE to the forms it specifies cleanly, while the > format has not shipped in a release. > > The changes: > > 1. Remove self-references. A value that sets offset and size with no uri is > invalid. offset may only be set together with uri. > 2. Resolve uri uniformly as an external reference, including when it names > the file that contains it. Parquet applies no compression or encryption of > its own to those bytes. The parquet writer does NOT own these bytes. An > earlier draft forbade referencing the current file, but neither a writer > nor a reader can check that, so the encoding is uniform instead. > 3. Drop the prohibition on modular encryption. It existed only because > self-referenced ranges were not encryption modules. The fields of a FILE > group are ordinary columns and are encoded, compressed, and encrypted like > any other column, inline included. > 4. Allow inline together with the locator fields. Both must denote the same > bytes, so a reader may resolve from either and get the same result, and > inline is the cheaper path because it needs no external access. If the two > disagree the value is invalid, and a reader is not required to detect it. > This combination was underspecified before. > > The spec changes are in https://github.com/apache/parquet-format/pull/603 > and touch LogicalTypes.md and a comment in parquet.thrift. Encryption.md is > unchanged from master and no Thrift structures change. Inline and external > references behave as before for existing readers and writers. > > Please vote in the next 72 hours. > > [ ] +1 Approve the proposed format change > [ ] +0 No opinion > [ ] -1 Do not approve (please provide specific reasons) > > Cheers, > > [1] https://github.com/apache/parquet-format/pull/603 > [2] https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7 >
