+1 (non-binding), thanks for clarifying this Alkis! Rok
On Mon, Aug 24, 2026 at 4:56 PM Divjot Arora via dev <[email protected]> wrote: > +1 (non-binding), thanks for pushing this forward! > > -- Div > > On Mon, Aug 24, 2026 at 7:51 AM Alkis Evlogimenos via dev < > [email protected]> wrote: > > > Thank you Russel. I updated the language to focus on reader semantics and > > saying that a reader can choose to read either. > > > > On Mon, Aug 24, 2026 at 4:37 PM Russell Spitzer < > [email protected] > > > > > wrote: > > > > > +1, Although I'm not sure we should add language about the bytes in the > > > location and inline *must* being the same. Since i'm not sure it's > > > something we could actually control from within the parquet universe. > If > > a > > > writer knows the two are identical at the time of writing the row is > that > > > enough? I think it's fine to outline the fallback behavior and just > say, > > > "When a locator is present, it is implied that a reader can either use > > the > > > locator or inline bytes interchangeably." > > > > > > On Mon, Aug 24, 2026 at 9:03 AM Alkis Evlogimenos via dev < > > > [email protected]> wrote: > > > > > >> Hi folks, > > >> > > >> I would like to start a vote on adopting the format change in PR 603 > > [1]: > > >> remove self-references from the FILE logical type. This came out of > the > > >> discussion on inheriting the inline codec [2]. > > >> > > >> FILE as merged lets a value set offset and size with no uri, which > > >> addresses a byte range inside the containing file. This byte range is > > >> effectively owned by the Parquet writer. Specifying that range > properly > > >> means giving it a compression block, an encryption module, an AAD > > >> identity, > > >> and its own size accounting. That is a second page mechanism reachable > > >> only > > >> through FILE. The use case is mid sized values and it is better served > > by > > >> non-contiguous pages, which we will propose separately. Removing > > >> self-references keeps FILE to the forms it specifies cleanly, while > the > > >> format has not shipped in a release. > > >> > > >> The changes: > > >> > > >> 1. Remove self-references. A value that sets offset and size with no > uri > > >> is > > >> invalid. offset may only be set together with uri. > > >> 2. Resolve uri uniformly as an external reference, including when it > > names > > >> the file that contains it. Parquet applies no compression or > encryption > > of > > >> its own to those bytes. The parquet writer does NOT own these bytes. > An > > >> earlier draft forbade referencing the current file, but neither a > writer > > >> nor a reader can check that, so the encoding is uniform instead. > > >> 3. Drop the prohibition on modular encryption. It existed only because > > >> self-referenced ranges were not encryption modules. The fields of a > FILE > > >> group are ordinary columns and are encoded, compressed, and encrypted > > like > > >> any other column, inline included. > > >> 4. Allow inline together with the locator fields. Both must denote the > > >> same > > >> bytes, so a reader may resolve from either and get the same result, > and > > >> inline is the cheaper path because it needs no external access. If the > > two > > >> disagree the value is invalid, and a reader is not required to detect > > it. > > >> This combination was underspecified before. > > >> > > >> The spec changes are in > > https://github.com/apache/parquet-format/pull/603 > > >> and touch LogicalTypes.md and a comment in parquet.thrift. > Encryption.md > > >> is > > >> unchanged from master and no Thrift structures change. Inline and > > external > > >> references behave as before for existing readers and writers. > > >> > > >> Please vote in the next 72 hours. > > >> > > >> [ ] +1 Approve the proposed format change > > >> [ ] +0 No opinion > > >> [ ] -1 Do not approve (please provide specific reasons) > > >> > > >> Cheers, > > >> > > >> [1] https://github.com/apache/parquet-format/pull/603 > > >> [2] https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7 > > >> > > > > > >
