winklemad opened a new pull request, #1219:
URL: https://github.com/apache/arrow-go/pull/1219
### Rationale for this change
`array.Concatenate` silently corrupts values when one of the inputs is a
`RunEndEncoded` array that was sliced in the **middle of a run**.
`updateRuns` normalizes each input's run ends by subtracting its logical
offset, but never clamps the final run end to the array's logical length. A
slice keeps the original physical end of the run it cuts through, so after
normalization the last run end overshoots the slice length. That overshoot then
shifts every following array's run ends, and the result still passes
`ValidateFull`, so nothing flags the corruption.
Reproduction (values are wrong, no error):
```go
b := array.NewRunEndEncodedBuilder(mem, arrow.PrimitiveTypes.Int32,
arrow.PrimitiveTypes.Int64)
vb := b.ValueBuilder().(*array.Int64Builder)
// run ends [3,5,8]: 100x3, 200x2, 300x3
b.Append(3); vb.Append(100); b.Append(2); vb.Append(200); b.Append(3);
vb.Append(300)
full := b.NewArray()
sliced := array.NewSlice(full, 1, 4) // logical [100,100,200], ends mid-run
of the 200s
b.Append(2); vb.Append(700)
tail := b.NewArray()
result, _ := array.Concatenate([]arrow.Array{sliced, tail}, mem)
// want [100,100,200,700,700]
// got [100,100,200,200,700] <- index 3 corrupted
```
`sliced` on its own decodes correctly (`[100,100,200]`); only the
concatenated result is wrong, so the defect is entirely in the run-end merge.
### What changes are included in this PR?
Clamp each input array's final run end to the running logical length in
`updateRuns` (`arrow/array/concat.go`). This is the single place run ends are
merged for `RunEndEncoded` concatenation (the generic function covers
int16/int32/int64 run-end types). No change to any array that ends on a run
boundary — only a slice that cuts through a run is affected, and it now stays
within its logical length.
### Are these changes tested?
Yes — added `TestConcatRunEndEncodedMidRunSlice` in
`arrow/array/concat_test.go`, which reproduces the corruption (it fails without
the fix) and uses a checked allocator to confirm no leaks. The existing
`TestConcatRunEndEncoded` / `TestConcatAlmostOverflowRunEndEncoding` and the
full `arrow/array` package tests still pass; `gofmt` and `go vet` are clean.
### Are there any user-facing changes?
Yes — `array.Concatenate` now returns correct values when an input is a
mid-run slice of a `RunEndEncoded` array, instead of silently wrong ones. No
API change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
