adriangb opened a new issue, #10896:
URL: https://github.com/apache/arrow-rs/issues/10896
**Is your feature request related to a problem or challenge?**
#10880 added `WriterProperties::resolve_column_properties`, which collapses
the
repeated `HashMap<ColumnPath, ColumnProperties>` lookups for a leaf column
into a
single lookup that returns all resolved settings at once.
That resolution currently happens in `GenericColumnWriter::new`
(`parquet/src/column/writer/mod.rs`). Column writers are recreated for every
row
group, so the cost is once per leaf column *per row group* rather than once
per
leaf column per file. For a wide schema written as many row groups — e.g.
10,000
columns × 100 row groups — that is still a million hash lookups over the
writer's
lifetime, even though every one of them returns the same answer.
**Describe the solution you'd like**
Resolve each leaf column's properties once when the file writer is
constructed and
reuse the result for every row group. Concretely: cache a
`Vec<ResolvedColumnProperties>` (indexed by leaf column) on
`SerializedFileWriter` / `ArrowWriter`, and thread the entry into
`get_column_writer` / `GenericColumnWriter::new` instead of having each
writer look
it up from `WriterProperties`. Resolution then becomes once per leaf column
per
file.
**Describe alternatives you've considered**
- Memoizing inside `WriterProperties` behind a lock — adds synchronization
to a
type that is currently shared freely via `WriterPropertiesPtr`, and still
costs a
lookup per row group.
- Leaving it as-is. The per-row-group cost is already far below the
pre-#10880
behaviour, so this is an incremental win rather than a fix; it matters
most for
wide-schema, many-row-group workloads.
**Additional context**
Follow-up from https://github.com/apache/arrow-rs/pull/10880 (see
https://github.com/apache/arrow-rs/pull/10880#issuecomment-5443041691).
`parquet/benches/writer_overhead.rs` already covers wide schemas; measuring
this
would want a variant that writes multiple row groups, since the existing
cases
write a single row group each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
