Hi everyone,
I'd like to share an update on the progress for the modular footer proposal [1]. We've received some valuable feedback and one recurring suggestion was to make the footer more future-proof by reusing Parquet encodings. We previously ruled out this approach because existing Parquet encodings don't support random access. However, with PFOR under development and potentially more random-access-friendly encodings coming in the future, we think it may be worth revisiting this idea. Another frequently raised point was compression, especially for cases where the footer becomes large enough that its size is a major bottleneck for cold reads. If we reuse Parquet pages to store footer metadata, we could also get compression support for free, potentially reducing footer size even further. At the same time, we want to make the modular footer flexible enough to accommodate future changes to the Parquet data layout. For example, the previously proposed non-contiguous pages layout [2] would require the optional offset index to be present, and we'd like to make its representation more efficient. Our goal is to design a footer that can support potential layout changes like this while still representing the current layout efficiently. We're still exploring these directions and will share updates as we learn more. In the meantime, please feel free to share any thoughts, concerns, or alternative ideas! Thanks again to everyone who reviewed the proposal and provided feedback! Best, Jiayi [1] https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma [2] https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > Hi everyone, > > We’ve been discussing the Modular Footer for quite a while, including > through a dedicated footer working group meeting every two weeks, and we > finally have a proposal ready for broader review. > > The proposal covers the motivation, format design, compatibility > considerations, and benchmark results. Huge thanks to everyone who has > contributed to the project and the discussions so far. > > The proposal doc, parquet-format PR, and tracking issue are all available > here: > > Proposal doc: > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing > > PR: https://github.com/apache/parquet-format/pull/623/changes > > Parquet-format issue: https://github.com/apache/parquet-format/issues/622 > > The proposal introduces several significant conceptual changes, so I > expect there will be a lot of discussion and feedback. For the first round > of review, I’d suggest focusing primarily on the high-level direction and > design choices, and leaving more detailed questions (for example, whether a > particular field can be deprecated) for later discussion. I think this will > help us converge on the major design questions first and keep the review > process moving efficiently. > > Thanks! > > Best, > Jiayi >
