Rich-T-kid opened a new issue, #11113: URL: https://github.com/apache/arrow-rs/issues/11113
**Is your feature request related to a problem or challenge?** Part of https://github.com/apache/arrow-rs/issues/7761 \`StructArray\` currently goes through \`GenericInProgressArray\` → \`concat_structs\`. \`concat_structs\` extracts each field across all buffered slices and calls \`concat()\` per field. This has two costs: 1. **2× peak memory per field**: all buffered struct slices and all field output arrays co-exist during the multi-field concat 2. **No recursive specialization**: each field's concat uses the concat kernel, which is fast for primitives and ByteView but still incurs the full batch-collection overhead **Describe the solution you'd like** \`InProgressStructArray\` maintains a \`Vec<Box<dyn InProgressArray>>\` — one per field — each using the optimal type-specific implementation: \`\`\`rust pub(crate) struct InProgressStructArray { fields: Fields, field_arrays: Vec<Box<dyn InProgressArray>>, // one per field nulls: NullBufferBuilder, source: Option<ArrayRef>, batch_size: usize, } \`\`\` **\`set_source(source)\` logic:** - Set \`self.source = source\` - For each field i: call \`field_arrays[i].set_source(Some(struct_arr.column(i).clone()))\` **\`copy_rows(offset, len)\` logic:** - Append outer nulls to \`NullBufferBuilder\` - For each field i: call \`field_arrays[i].copy_rows(offset, len)\` **\`finish()\` logic:** - For each field i: call \`field_arrays[i].finish()\` → collect into \`Vec<ArrayRef>\` - Build \`StructArray::try_new_with_length(fields, columns, nulls, len)\` **Why better than concat:** - Each field builds incrementally → no 2× peak memory - Each field automatically inherits its optimal InProgressArray (e.g., \`Float64\` field → \`InProgressPrimitiveArray\` with pre-allocated typed vec; \`Utf8View\` field → \`InProgressByteViewArray\` with buffer reuse) - For structs with many primitive fields (the common case in DataFusion record batches), this compounds the benefit across all fields **\`create_in_progress_array\` dispatch:** \`\`\`rust DataType::Struct(fields) => { let field_arrays = fields.iter() .map(|f| create_in_progress_array(f.data_type(), batch_size)) .collect(); Box::new(InProgressStructArray::new(fields.clone(), field_arrays, batch_size)) } \`\`\` **Benchmarks** Add struct schemas (struct of primitives, struct with StringView fields, deeply nested structs): \`\`\` cargo bench --bench coalesce_kernels --features test_utils -- struct \`\`\` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
