+1 (non-binding), thanks for pushing this forward! -- Div
On Mon, Aug 24, 2026 at 7:51 AM Alkis Evlogimenos via dev < [email protected]> wrote: > Thank you Russel. I updated the language to focus on reader semantics and > saying that a reader can choose to read either. > > On Mon, Aug 24, 2026 at 4:37 PM Russell Spitzer <[email protected] > > > wrote: > > > +1, Although I'm not sure we should add language about the bytes in the > > location and inline *must* being the same. Since i'm not sure it's > > something we could actually control from within the parquet universe. If > a > > writer knows the two are identical at the time of writing the row is that > > enough? I think it's fine to outline the fallback behavior and just say, > > "When a locator is present, it is implied that a reader can either use > the > > locator or inline bytes interchangeably." > > > > On Mon, Aug 24, 2026 at 9:03 AM Alkis Evlogimenos via dev < > > [email protected]> wrote: > > > >> Hi folks, > >> > >> I would like to start a vote on adopting the format change in PR 603 > [1]: > >> remove self-references from the FILE logical type. This came out of the > >> discussion on inheriting the inline codec [2]. > >> > >> FILE as merged lets a value set offset and size with no uri, which > >> addresses a byte range inside the containing file. This byte range is > >> effectively owned by the Parquet writer. Specifying that range properly > >> means giving it a compression block, an encryption module, an AAD > >> identity, > >> and its own size accounting. That is a second page mechanism reachable > >> only > >> through FILE. The use case is mid sized values and it is better served > by > >> non-contiguous pages, which we will propose separately. Removing > >> self-references keeps FILE to the forms it specifies cleanly, while the > >> format has not shipped in a release. > >> > >> The changes: > >> > >> 1. Remove self-references. A value that sets offset and size with no uri > >> is > >> invalid. offset may only be set together with uri. > >> 2. Resolve uri uniformly as an external reference, including when it > names > >> the file that contains it. Parquet applies no compression or encryption > of > >> its own to those bytes. The parquet writer does NOT own these bytes. An > >> earlier draft forbade referencing the current file, but neither a writer > >> nor a reader can check that, so the encoding is uniform instead. > >> 3. Drop the prohibition on modular encryption. It existed only because > >> self-referenced ranges were not encryption modules. The fields of a FILE > >> group are ordinary columns and are encoded, compressed, and encrypted > like > >> any other column, inline included. > >> 4. Allow inline together with the locator fields. Both must denote the > >> same > >> bytes, so a reader may resolve from either and get the same result, and > >> inline is the cheaper path because it needs no external access. If the > two > >> disagree the value is invalid, and a reader is not required to detect > it. > >> This combination was underspecified before. > >> > >> The spec changes are in > https://github.com/apache/parquet-format/pull/603 > >> and touch LogicalTypes.md and a comment in parquet.thrift. Encryption.md > >> is > >> unchanged from master and no Thrift structures change. Inline and > external > >> references behave as before for existing readers and writers. > >> > >> Please vote in the next 72 hours. > >> > >> [ ] +1 Approve the proposed format change > >> [ ] +0 No opinion > >> [ ] -1 Do not approve (please provide specific reasons) > >> > >> Cheers, > >> > >> [1] https://github.com/apache/parquet-format/pull/603 > >> [2] https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7 > >> > > >
