Rich-T-kid opened a new pull request, #11120: URL: https://github.com/apache/arrow-rs/pull/11120
## What Adds `InProgressFixedSizeListArray`, a specialized `InProgressArray` for `FixedSizeList(T, N)` columns in `BatchCoalescer`. ## Why Previously, `FixedSizeList` went through `GenericInProgressArray` → `concat_fallback` → `MutableArrayData`. The MutableArrayData path recursively extends child arrays also through MutableArrayData, even when the child type (e.g. `Float32`) has a faster specialized path. This PR: 1. Builds the outer null bitmap incrementally via `NullBufferBuilder` 2. Buffers child value slices (zero-copy from the source array for the non-filter path) 3. At `finish()`, calls `concat()` on the child slices, which dispatches to the type-optimized path (e.g. `concat_primitives` for `Float32` child arrays) Common use case: embedding vectors (`FixedSizeList<Float32, 128>`) used in ML pipelines; also geospatial coords (size=2/3/4). ## Benchmarks Run with `cargo bench --bench coalesce_kernels --features test_utils -- fsl`. | Benchmark | Before | After | Speedup | |---|---|---|---| | fsl(4), no nulls, sel=0.001 | 81.1 ms | 63.3 ms | **−22%** | | fsl(128), no nulls, sel=0.001 | 97.5 ms | 76.0 ms | **−22%** | | fsl(4), no nulls, sel=0.01 | 11.2 ms | 9.2 ms | **−18%** | | fsl(128), no nulls, sel=0.01 | 21.9 ms | 19.1 ms | **−13%** | | fsl(4), no nulls, sel=0.1 | 5.3 ms | 4.8 ms | **−10%** | | fsl(128), no nulls, sel=0.1 | 16.1 ms | 13.5 ms | **−16%** | | fsl(4), no nulls, sel=0.8 | 1.31 ms | 1.16 ms | **−12%** | | fsl(128), no nulls, sel=0.8 | 9.2 ms | 8.0 ms | **−13%** | | fsl(4), nulls=0.1, sel=0.001 | 180.6 ms | 73.3 ms | **−59%** | | fsl(128), nulls=0.1, sel=0.001 | 117.2 ms | 84.9 ms | **−27%** | | fsl(4), nulls=0.1, sel=0.1 | 7.5 ms | 6.5 ms | **−14%** | | fsl(128), nulls=0.1, sel=0.1 | 56.3 ms | 16.7 ms | **−70%** | | fsl(4), nulls=0.1, sel=0.8 | 4.4 ms | 1.72 ms | **−61%** | | fsl(128), nulls=0.1, sel=0.8 | 15.9 ms | 9.4 ms | **−41%** | The nullable cases see the largest gains (up to 70%) because the old path was hitting a particularly slow codepath through MutableArrayData when nulls were present. Part of https://github.com/apache/arrow-rs/issues/7761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
