Hey everyone, I wanted to follow up on the progress and path forward on this proposal.
There's been some discussion about the scope of this change (both in the doc and amongst those looking into this) and we identified a few separate issues that we want to break into different proposals. What is described in the proposal doc primarily addresses the physical layout of pages and we want to preserve this as the main focus of this effort. Separately, there are two other concerns that can be addressed in parallel, but do not need to be coupled with this proposal: 1. Granularity of stats (row oriented stats vs page oriented stats): The intent here is to separate stats from the physical layout so they can be controlled independently. 2. Dictionary fallback: The intent here is to improve dictionary effectiveness and allow multiple dictionaries within a column >From here, we'll be looking at a more complete implementation of non-contiguous pages that address some of the concerns raised about fragmentation and performance with this new layout. -Dan On Wed, May 6, 2026 at 1:46 PM Andrew Lamb <[email protected]> wrote: > Thank you -- I really like the idea and left some comments. > > > The core problem is writer memory pressure caused by wide schemas and > asymmetric column sizes. T > > I think another important problem that this proposal addresses is that the > current layout can force scattered reads when doing point lookups, which > are increasingly important for AI workloads. > > Weston Pace, from LanceDB, also gave talk recently[1] about challenges > using Parquet that basically describe these issues, in case anyone would > like a higher bandwidth (and amusing) take on the challenge > > https://www.youtube.com/watch?v=fDrmfDuPK3s > > Andrew > > > On Wed, May 6, 2026 at 3:27 AM Will Edwards via dev < > [email protected]> > wrote: > > > Hi Daniel, > > > > Interesting problem, it's good that we are thinking about this. > > > > Memory pressure is definitely a problem, particularly for row-wise > writers > > that must buffer all the column chunks for the row group before writing > > them. > > > > Trivia: Most open-source column-wise writers I've seen also buffer all > the > > column chunks for the row group before writing them, although they > strictly > > don't need to. DuckDB has a really interesting hybrid that serializes > > eight columns at a time, which I'd really like to understand better. > > Arrow-cpp's WriteTable processes one column at a time, offering memory > > advantages. > > > > Also, many modern writers split into row groups based on row count > instead > > of byte size; some use compressed size, while others use uncompressed > > size. They generally aren't ready for very large values in certain > columns > > within some rows. > > > > Of course the row-group-size is not just about the writer's memory > > management; it's a contract with the reader regarding the memory > > requirements the reader will need. Parquet writers that generate very > > large row groups efficiently can easily create Parquet files that even > > mainstream readers with ample RAM cannot manage to read. > > > > One implementation detail concerns using `data_page_offset = -1` as a > > marker. Most open source readers I've seen fail (often with an I/O > error) > > if you access that column. Arrow rust will actually panic if any footer > > columns have data_page_offset = -1, even if that column isn't in the > > projection (and panic can map to sigabrt on Rust; it's a compiler > > setting). DuckDB silently corrupts data. > > > > Other approaches might exist. For example, there might be a new > DataPageV3 > > (yes, the point is for readers who don't know to fail when they encounter > > it!). DataPageV3 doesn't store data inline, instead, it contains the > > OffsetIndex. (The expectation that OffsetIndex sits outside row > > groups—which seems a strange insistence in the spec—is relaxed). Engines > > unaware of DataPageV3 that use offset indexes to always seek will get the > > real data. Most don't, though, so fail. > > > > It's easy to imagine future Parquet files that have one big row group and > > discontinuous column chunks. It's basically half way to Lance? > > > > Many engines today pay no attention to page indexes, or if they do, they > > use them for stats rather than for seeking. The new discontinuous column > > chunks might force a significant architectural change for those engines. > > > > Fun thinking about this kind of stuff, > > Will > > > > > > On Tue, 5 May 2026 at 01:18, Daniel Weeks <[email protected]> > > wrote: > > > > > Hey Parquet Devs, > > > > > > I would like to introduce a proposal that addresses the issues arising > > from > > > the physical layout requirements in the Parquet format that necessitate > > > contiguous data for columnar data. > > > > > > Over the years, several improvements were introduced to solve other > > > challenges, effectively capturing the necessary information for Parquet > > to > > > lift the contiguity requirement on pages and column chunks. > > > > > > Other formats recognize these challenges and embrace a model where > > > individual column segments are tracked at the metadata level but do not > > > rely on physical contiguity in the file. > > > > > > The core problem is writer memory pressure caused by wide schemas and > > > asymmetric column sizes. Today a writer must buffer every column chunk > in > > > memory until a row group is complete, because each column chunk must > land > > > as a single contiguous byte range. For wide schemas, or schemas mixing > > > small fixed-width columns with very large variable-length values, this > > can > > > drive high memory usage even when individual pages are fully encoded, > > > compressed, and ready to flush, or it can result in row groups being > > > produced at inconsistent or inefficient boundaries. > > > > > > This characteristic is more pronounced for emerging AI/ML use cases > that > > > rely on data types and sizes atypical for traditional analytic use > cases. > > > > > > The document linked below includes a comprehensive proposal. Looking > > > forward to your feedback. > > > > > > Proposal: > > > > > > > > > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA > > > > > > Thanks, > > > Dan > > > > > >
