Hi everyone, As mentioned previously, we have revisited the proposal [1] to make it more flexible and also address existing feedback. I’d like to share the new version of the proposal [2], which is included as a new tab in the same document. A summary of what changed and why: Flexible Data Layouts
Per discussions on evolving Parquet data layout (e.g. the “non-contiguous pages” proposal), our goal is to have the new footer handle both existing and potential future layouts seamlessly. The updated proposal doesn’t assume that row groups are physical boundaries. It preserves the wording of “column chunks”, but these are simply defined to mean a contiguous set of pages for one column. Different columns may have different numbers of chunks in a file. The “row group” wording is preserved as well, but row group metadata is made optional. Lastly, stats are re-defined based on row ranges rather than row groups; different columns may publish stats at different granularities. Page-based Representation A big feedback point on the initial proposal was that the EncodedArray concept was opaque/confusing. It was added to efficiently model arrays in Thrift and provide random access, at the expense of readability. The updated proposal uses Parquet pages to represent modules, which yields a self-describing layout that evolves naturally as Parquet gains new encodings. It also allows for optional per-page compression. Overall, the new proposal yields a footer that is comparable in size and performance to the original proposal, but is simpler and more flexible for handling future Parquet layout changes. We appreciate any feedback on the proposal. Best, Jiayi [1] https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma#heading=h.kfs863ye4rzw [2] https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.cbl5qtgc5xhz Jiayi Wang <[email protected]> 于2026年10月19日周一 20:49写道： > Hi everyone, > > > I'd like to share an update on the progress for the modular footer > proposal [1]. We've received some valuable feedback and one recurring > suggestion was to make the footer more future-proof by reusing Parquet > encodings. > > > We previously ruled out this approach because existing Parquet encodings > don't support random access. However, with PFOR under development and > potentially more random-access-friendly encodings coming in the future, we > think it may be worth revisiting this idea. > > > Another frequently raised point was compression, especially for cases > where the footer becomes large enough that its size is a major bottleneck > for cold reads. If we reuse Parquet pages to store footer metadata, we > could also get compression support for free, potentially reducing footer > size even further. > > > At the same time, we want to make the modular footer flexible enough to > accommodate future changes to the Parquet data layout. For example, the > previously proposed non-contiguous pages layout [2] would require the > optional offset index to be present, and we'd like to make its > representation more efficient. Our goal is to design a footer that can > support potential layout changes like this while still representing the > current layout efficiently. > > > We're still exploring these directions and will share updates as we learn > more. In the meantime, please feel free to share any thoughts, concerns, or > alternative ideas! > > > Thanks again to everyone who reviewed the proposal and provided feedback! > > > Best, > Jiayi > > > [1] > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma > > [2] > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 > > Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > >> Hi everyone, >> >> We’ve been discussing the Modular Footer for quite a while, including >> through a dedicated footer working group meeting every two weeks, and we >> finally have a proposal ready for broader review. >> >> The proposal covers the motivation, format design, compatibility >> considerations, and benchmark results. Huge thanks to everyone who has >> contributed to the project and the discussions so far. >> >> The proposal doc, parquet-format PR, and tracking issue are all available >> here: >> >> Proposal doc: >> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing >> >> PR: https://github.com/apache/parquet-format/pull/623/changes >> >> Parquet-format issue: https://github.com/apache/parquet-format/issues/622 >> >> The proposal introduces several significant conceptual changes, so I >> expect there will be a lot of discussion and feedback. For the first round >> of review, I’d suggest focusing primarily on the high-level direction and >> design choices, and leaving more detailed questions (for example, whether a >> particular field can be deprecated) for later discussion. I think this will >> help us converge on the major design questions first and keep the review >> process moving efficiently. >> >> Thanks! >> >> Best, >> Jiayi >> > Jiayi Wang <[email protected]> 于2026年10月19日周一 20:49写道： > Hi everyone, > > > I'd like to share an update on the progress for the modular footer > proposal [1]. We've received some valuable feedback and one recurring > suggestion was to make the footer more future-proof by reusing Parquet > encodings. > > > We previously ruled out this approach because existing Parquet encodings > don't support random access. However, with PFOR under development and > potentially more random-access-friendly encodings coming in the future, we > think it may be worth revisiting this idea. > > > Another frequently raised point was compression, especially for cases > where the footer becomes large enough that its size is a major bottleneck > for cold reads. If we reuse Parquet pages to store footer metadata, we > could also get compression support for free, potentially reducing footer > size even further. > > > At the same time, we want to make the modular footer flexible enough to > accommodate future changes to the Parquet data layout. For example, the > previously proposed non-contiguous pages layout [2] would require the > optional offset index to be present, and we'd like to make its > representation more efficient. Our goal is to design a footer that can > support potential layout changes like this while still representing the > current layout efficiently. > > > We're still exploring these directions and will share updates as we learn > more. In the meantime, please feel free to share any thoughts, concerns, or > alternative ideas! > > > Thanks again to everyone who reviewed the proposal and provided feedback! > > > Best, > Jiayi > > > [1] > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma > > [2] > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 > > Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > >> Hi everyone, >> >> We’ve been discussing the Modular Footer for quite a while, including >> through a dedicated footer working group meeting every two weeks, and we >> finally have a proposal ready for broader review. >> >> The proposal covers the motivation, format design, compatibility >> considerations, and benchmark results. Huge thanks to everyone who has >> contributed to the project and the discussions so far. >> >> The proposal doc, parquet-format PR, and tracking issue are all available >> here: >> >> Proposal doc: >> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing >> >> PR: https://github.com/apache/parquet-format/pull/623/changes >> >> Parquet-format issue: https://github.com/apache/parquet-format/issues/622 >> >> The proposal introduces several significant conceptual changes, so I >> expect there will be a lot of discussion and feedback. For the first round >> of review, I’d suggest focusing primarily on the high-level direction and >> design choices, and leaving more detailed questions (for example, whether a >> particular field can be deprecated) for later discussion. I think this will >> help us converge on the major design questions first and keep the review >> process moving efficiently. >> >> Thanks! >> >> Best, >> Jiayi >> >
