alamb opened a new issue, #8389:
URL: https://github.com/apache/arrow-rs/issues/8389
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
We now have two different APIs for writing row groups in parallel,
depending on encryption, and I would like to simplify the code to use just one.
The current
[example](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html#example-encoding-two-arrow-arrays-in-parallel)
for writing row groups in parallel uses
[`get_column_writers`](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/fn.get_column_writers.html)
and does not support encryption
@rok and @adamreeve added a new API based on `ArrowRowGroupWriterFactory`
for encoding parquet columns and row groups in parallel, with encryption in
- https://github.com/apache/arrow-rs/pull/8162,
This API is also somewhat strange in that it makes users create an
`ArrowWriter` only to immediately destructure it into a `SerializedWriter` /
the underlying writer.
The reason we need to expose `ArrowRowGroupWriterFactory` is that
`ArrowRowGroupWriterFactory::create_column_writers` also has the appropriate
encryption properties whereas `get_column_writers` does not
**Describe the solution you'd like**
I would like a single easy to use API for writing in parallel that:
1. Is the same for encryption vs not encryption
2. Has clear examples
**Describe alternatives you've considered**
I suggest:
1. Make the constructors for `ArrowRowGroupWriterFactory` public
2. Update the
[example](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowColumnWriter.html#example-encoding-two-arrow-arrays-in-parallel)
to use `ArrowRowGroupWriterFactory` /
`ArrowRowGroupWriterFactory::create_column_writers` function
3. Deprecating the existing
[`get_column_writers`](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/fn.get_column_writers.html)
function directing people to `ArrowRowGroupWriterFactory`
4. Deprecate `ArrowWriter::into_serialized_writer`, directing people to
`ArrowRowGroupWriterFactory`
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
