Thank you Jiayi for leading this effort and everyone who contributed. I would encourage folks on the list to review as it is fairly transformational for the project. I expect that after that we'll avoid changing the footer format again for a while.
On Fri, Oct 30, 2026 at 7:20 AM Jiayi Wang <[email protected]> wrote: > Hi everyone, > > As mentioned previously, we have revisited the proposal [1] to make it > more flexible and also address existing feedback. I’d like to share the new > version of the proposal [2], which is included as a new tab in the same > document. A summary of what changed and why: > Flexible Data Layouts > > Per discussions on evolving Parquet data layout (e.g. the “non-contiguous > pages” proposal), our goal is to have the new footer handle both existing > and potential future layouts seamlessly. The updated proposal doesn’t > assume that row groups are physical boundaries. It preserves the wording of > “column chunks”, but these are simply defined to mean a contiguous set of > pages for one column. Different columns may have different numbers of > chunks in a file. The “row group” wording is preserved as well, but row > group metadata is made optional. Lastly, stats are re-defined based on row > ranges rather than row groups; different columns may publish stats at > different granularities. > Page-based Representation > > A big feedback point on the initial proposal was that the EncodedArray > concept was opaque/confusing. It was added to efficiently model arrays in > Thrift and provide random access, at the expense of readability. The > updated proposal uses Parquet pages to represent modules, which yields a > self-describing layout that evolves naturally as Parquet gains new > encodings. It also allows for optional per-page compression. > > Overall, the new proposal yields a footer that is comparable in size and > performance to the original proposal, but is simpler and more flexible for > handling future Parquet layout changes. We appreciate any feedback on the > proposal. > > Best, > > Jiayi > > [1] > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma#heading=h.kfs863ye4rzw > > [2] > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.cbl5qtgc5xhz > > > > Jiayi Wang <[email protected]> 于2026年10月19日周一 20:49写道： > > > Hi everyone, > > > > > > I'd like to share an update on the progress for the modular footer > > proposal [1]. We've received some valuable feedback and one recurring > > suggestion was to make the footer more future-proof by reusing Parquet > > encodings. > > > > > > We previously ruled out this approach because existing Parquet encodings > > don't support random access. However, with PFOR under development and > > potentially more random-access-friendly encodings coming in the future, > we > > think it may be worth revisiting this idea. > > > > > > Another frequently raised point was compression, especially for cases > > where the footer becomes large enough that its size is a major bottleneck > > for cold reads. If we reuse Parquet pages to store footer metadata, we > > could also get compression support for free, potentially reducing footer > > size even further. > > > > > > At the same time, we want to make the modular footer flexible enough to > > accommodate future changes to the Parquet data layout. For example, the > > previously proposed non-contiguous pages layout [2] would require the > > optional offset index to be present, and we'd like to make its > > representation more efficient. Our goal is to design a footer that can > > support potential layout changes like this while still representing the > > current layout efficiently. > > > > > > We're still exploring these directions and will share updates as we learn > > more. In the meantime, please feel free to share any thoughts, concerns, > or > > alternative ideas! > > > > > > Thanks again to everyone who reviewed the proposal and provided feedback! > > > > > > Best, > > Jiayi > > > > > > [1] > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma > > > > [2] > > > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 > > > > Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > > > >> Hi everyone, > >> > >> We’ve been discussing the Modular Footer for quite a while, including > >> through a dedicated footer working group meeting every two weeks, and we > >> finally have a proposal ready for broader review. > >> > >> The proposal covers the motivation, format design, compatibility > >> considerations, and benchmark results. Huge thanks to everyone who has > >> contributed to the project and the discussions so far. > >> > >> The proposal doc, parquet-format PR, and tracking issue are all > available > >> here: > >> > >> Proposal doc: > >> > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing > >> > >> PR: https://github.com/apache/parquet-format/pull/623/changes > >> > >> Parquet-format issue: > https://github.com/apache/parquet-format/issues/622 > >> > >> The proposal introduces several significant conceptual changes, so I > >> expect there will be a lot of discussion and feedback. For the first > round > >> of review, I’d suggest focusing primarily on the high-level direction > and > >> design choices, and leaving more detailed questions (for example, > whether a > >> particular field can be deprecated) for later discussion. I think this > will > >> help us converge on the major design questions first and keep the review > >> process moving efficiently. > >> > >> Thanks! > >> > >> Best, > >> Jiayi > >> > > > Jiayi Wang <[email protected]> 于2026年10月19日周一 20:49写道： > > > Hi everyone, > > > > > > I'd like to share an update on the progress for the modular footer > > proposal [1]. We've received some valuable feedback and one recurring > > suggestion was to make the footer more future-proof by reusing Parquet > > encodings. > > > > > > We previously ruled out this approach because existing Parquet encodings > > don't support random access. However, with PFOR under development and > > potentially more random-access-friendly encodings coming in the future, > we > > think it may be worth revisiting this idea. > > > > > > Another frequently raised point was compression, especially for cases > > where the footer becomes large enough that its size is a major bottleneck > > for cold reads. If we reuse Parquet pages to store footer metadata, we > > could also get compression support for free, potentially reducing footer > > size even further. > > > > > > At the same time, we want to make the modular footer flexible enough to > > accommodate future changes to the Parquet data layout. For example, the > > previously proposed non-contiguous pages layout [2] would require the > > optional offset index to be present, and we'd like to make its > > representation more efficient. Our goal is to design a footer that can > > support potential layout changes like this while still representing the > > current layout efficiently. > > > > > > We're still exploring these directions and will share updates as we learn > > more. In the meantime, please feel free to share any thoughts, concerns, > or > > alternative ideas! > > > > > > Thanks again to everyone who reviewed the proposal and provided feedback! > > > > > > Best, > > Jiayi > > > > > > [1] > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma > > > > [2] > > > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 > > > > Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > > > >> Hi everyone, > >> > >> We’ve been discussing the Modular Footer for quite a while, including > >> through a dedicated footer working group meeting every two weeks, and we > >> finally have a proposal ready for broader review. > >> > >> The proposal covers the motivation, format design, compatibility > >> considerations, and benchmark results. Huge thanks to everyone who has > >> contributed to the project and the discussions so far. > >> > >> The proposal doc, parquet-format PR, and tracking issue are all > available > >> here: > >> > >> Proposal doc: > >> > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing > >> > >> PR: https://github.com/apache/parquet-format/pull/623/changes > >> > >> Parquet-format issue: > https://github.com/apache/parquet-format/issues/622 > >> > >> The proposal introduces several significant conceptual changes, so I > >> expect there will be a lot of discussion and feedback. For the first > round > >> of review, I’d suggest focusing primarily on the high-level direction > and > >> design choices, and leaving more detailed questions (for example, > whether a > >> particular field can be deprecated) for later discussion. I think this > will > >> help us converge on the major design questions first and keep the review > >> process moving efficiently. > >> > >> Thanks! > >> > >> Best, > >> Jiayi > >> > > >
