rabenhorst opened a new issue, #12072:
URL: https://github.com/apache/arrow-rs/issues/12072
### Is your feature request related to a problem or challenge?
`SerializedFileWriter` keeps three `Vec<Option<_>>` per row group until the
file is
closed: bloom filters (`Sbbf`), column indexes (`ColumnIndexMetaData`) and
offset
indexes (`OffsetIndexMetaData`), each with one entry per leaf column. They
are kept
even when every entry is `None`, for example when column chunks are copied
with
`SerializedRowGroupWriter::append_column` without indexes, when page
statistics are
disabled, or after bloom filters were already written with
`BloomFilterPosition::AfterRowGroup`.
On a 64-bit target the three element types take 24 + 232 + 48 = 304 bytes,
so the
writer holds 304 bytes per leaf column per row group that carry no
information. For a
file with 200 leaf columns and 10,000 row groups that is 608 MB held until
`close`,
about 40% of the writer's heap at that point (1.52 GB). This shows up when
many small
files are concatenated into one large file with `append_column`.
### Describe the solution you'd like
Store `None` instead of a per-column vector for a row group that has no
bloom filter,
column index or offset index entry, and expand it to the dense layout only
where that
is needed: the `PageIndex` passed to the metadata writer and returned in
`ParquetMetaData`. Written bytes, the returned `ParquetMetaData` and the
public API stay
the same (`OnCloseRowGroup` still receives one entry per column).
### Describe alternatives you've considered
- Callers that write no indexes can call `set_offset_index_disabled(true)`
(with
statistics below `Page`, otherwise the setting is overridden). That avoids
the dense
offset index in the returned metadata, but not the vectors retained while
writing.
- A streaming footer would bound metadata memory independently of the number
of row
groups, but that is a much larger change.
### Additional context
I have a change for this ready and will open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
