HippoBaro commented on code in PR #11278:
URL: https://github.com/apache/arrow-rs/pull/11278#discussion_r4174936398
##########
parquet/src/arrow/arrow_writer/levels.rs:
##########
@@ -208,16 +230,19 @@ impl LevelInfoBuilder {
let levels = ArrayLevels::new(parent_ctx, is_nullable,
array.clone());
Ok(Self::Primitive(levels))
}
- DataType::RunEndEncoded(_, value_field) => {
+ DataType::RunEndEncoded(_, _) => {
let flat = expand_ree_array(array)?;
- let flat_field = Field::new(
- field.name(),
- value_field.data_type().clone(),
- field.is_nullable(),
- );
- Self::try_new(&flat_field, parent_ctx, &flat)
+ Self::try_new(field, parent_ctx, &flat)
}
- DataType::Struct(children) => {
+ DataType::Dictionary(_, _) => {
Review Comment:
Yep, worth measuring. Scalar dictionaries keep the existing leaf path, so
they don’t pay anything there. We don’t currently have a benchmark that hits
this particular branch.
One useful bit of context: the native REE work in #11273 already removes
this fallback. The replacement follows dictionary keys into the original nested
values rather than materializing an expanded copy. So this is a
correctness-first bridge for the current writer, not the intended long-term
path. That doesn’t make the interim cost free though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
