Hi all!
Following up on the Column File representation thread
<https://lists.apache.org/thread/jbh1gbrso5h6l4by9rh9poy2cjjtb8j0>, I'd like to
fire off a discussion about a possible optimization for Column Updates, where
supporting writers might decide to write out the Column File aligning all row
groups with the Base File, enabling supporting readers to do simple zero-copy
reads.
*Context*
The original thread settled on a dense representation (i.e. Column Files contain
exactly the same amount of rows as Base Files), while allowing unsynchronized
row-group boundaries.
I'm proposing adding a separate metadata flag (e.g.
`ColumnFile.containsAlignedRowGroups`) which, when set, indicates that the row
groups contained in the associated Column File are aligned with the Base File,
and allows readers to directly swap a Base File column chunk with the Column
File one on byte level.
By "aligned" here I mean that if a Base File contains two row groups with rows
e.g. 0-1000 and 1001-2000, the Column File contains two row groups with exactly
the same row boundaries 0-1000 and 1001-2000; while "unaligned" allows the
Column File to contain row boundaries e.g. 0-1500 and 1501-2000.
*Performance benefits*
Assuming a somewhat uniform distrbution of row group byte sizes (say, RG is the
average row group size in bytes), reading a single Column File incurs anywhere
from 0 (row groups already match by chance) to 2*RG (one row before, one row
after) bytes of overhead I/O and memory that is being discarded by the reader.
Allowing always aligning row group boundaries allows readers to lower that
overhead cost to exactly zero.
*Pros:*
- Writers are not required to implement -- keeping the flag false is still
valid for any data shape.
- Readers are not required to implement -- aligning row groups doesn't break
readers that still want to do stitching on row level.
- One less copy of data along the way from parquet to executor.
- Less reader I/O.
*Cons:*
- Implementations are optional -- leads to both diverging features in
different implementations on one hand, and diverging codepaths within a
single Iceberg implementation as support for the unaligned case is still
required.
- One more field in the Column File struct -- more support work.
Thanks!
Leonid.
