Rich-T-kid opened a new issue, #11112: URL: https://github.com/apache/arrow-rs/issues/11112
**Is your feature request related to a problem or challenge?** Part of https://github.com/apache/arrow-rs/issues/7761 \`RunEndEncoded\` arrays currently go through \`GenericInProgressArray\` → \`concat_run_arrays\`. While \`concat_run_arrays\` is a specialized concat function (not MutableArrayData), a custom \`InProgressArray\` can do something \`concat\` fundamentally cannot: **merge adjacent runs at batch boundaries**. When batch N ends with value \`"A"\` and batch N+1 begins with value \`"A"\`, these form a single logical run that \`concat_run_arrays\` splits across a batch boundary, inflating the run count. A specialized implementation can detect and merge these adjacent equal runs. **Describe the solution you'd like** \`InProgressRunEndEncodedArray\` with two internal components: 1. **\`run_ends: Vec<RunEndNative>\`** — accumulated run ends, adjusted to logical absolute positions as each batch is pushed 2. **\`values: Box<dyn InProgressArray>\`** — the values child uses its own optimal InProgressArray (e.g. \`InProgressPrimitiveArray\` if values are Int32, \`InProgressByteViewArray\` if StringView) **\`copy_rows(offset, len)\` logic:** - Walk the source REE's run structure for the [offset, offset+len range - For each run in range: compute its adjusted run_end and push to `run_ends` - **Boundary merge**: before pushing the first run, check if the last accumulated value equals the first incoming run's value → if equal, extend the last `run_end` instead of adding a new entry - Push values through the child `InProgressArray` **`finish()` logic:** - Finish the values child → get values `ArrayRef` - Build `run_ends` as a `PrimitiveArray<R>` - Construct `RunArray<R>` directly from run_ends + values **Why better than concat:** - `concat_run_arrays` processes batches independently and cannot merge cross-batch runs → the output may have `N-1` unnecessary run boundaries for `N` batches - In sorted data patterns (WHERE clause on a sorted column), an entire batch may be the same value → cross-batch merging eliminates entire runs - The values child benefits from its own type-specific InProgressArray (incremental, no 2× memory) **Benchmarks** Add benchmarks with sorted/grouped data that exercises the run-merging case: ``` cargo bench --bench coalesce_kernels --features test_utils -- run_end ``` EOF ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
