I wanted to follow up based on the discussion in the parquet sync and highlight a few things:
(Notes summary here <https://docs.google.com/document/d/11auIypU3ZfUvuusyWm1SLTFodnUsdf3b-jpG_R4MmKc/edit?tab=t.xf2812q29vzd#bookmark=id.xgdbdak3iq3p>, transcript here <https://docs.google.com/document/d/11auIypU3ZfUvuusyWm1SLTFodnUsdf3b-jpG_R4MmKc/edit?tab=t.rzuip2y0am75#heading=h.22t1qst27vv3> ) One of the nice features of the additional inline and self-referencing path is that the read path is trivial regardless of where the data is stored. This leaves the complexity up to the writer implementations to decide which approach they want to use, but consumers aren't burdened by those choices. Other than that, the proposal is still very much inline with the principals we discussed before about keeping the type minimal. I think the next steps from here are to update the PR to reflect these changes and start making some progress on reference implementations. I'd love to hear if there is disagreement with this as a path forward. Thanks, Dan On Tue, Jun 30, 2026 at 8:37 AM Alkis Evlogimenos via dev < [email protected]> wrote: > Hello, > > The FILE PR [1] is in good shape — path/size/offset/etag are in, and > there's an active thread on adding content_type. Following last week's sync > and the discussing in the comments of the PR, we took a step back and > looked at the difference in complexity between having two separate types: > one narrowly focused on referencing a file and later introducing one that > can accommodate inline/in-file/packed representations and it all came down > to the addition of a single field (along with how fields are interpreted). > Rather than have multiple types that significantly overlap in capability > and representation, we think it make sense to provide a single type that > can address all necessary features. The changes in short: > > - make path optional — an absent path means a self-reference to the > current file (already raised in review; makes self-referencing files > trivially moveable) > - add inline for storing bytes directly in the value > - generalize etag into a self-describing checksum (it carries an etag as > one case) > > FILE then spans inline/in-file/external as one type. Defining the full set > now avoids a second type later that every reader and writer would have to > handle. > > Details in the "FILE — complete type" tab of the design doc [2]. Feedback > welcome. > > Regards, > Alkis > > [1] https://github.com/apache/parquet-format/pull/585 > [2] > > https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit >
