rustyconover opened a new issue, #462:
URL: https://github.com/apache/arrow-dotnet/issues/462
## Summary
`ArrowArrayConcatenator.Concatenate` throws `ArgumentOutOfRangeException`
when one of the inputs is a zero-length `string`, `binary`, `large_string` or
`large_binary` array whose offsets buffer is empty (0 bytes).
That input is valid, and it arrives through IPC. For a length-0
variable-size array, Arrow C++ accepts an absent or empty offsets buffer
(ARROW-544, `ValidateOffsetsAndSizes` in `cpp/src/arrow/array/validate.cc`:
"For length 0, an empty offsets buffer is accepted"). Its IPC writer emits that
buffer as it is (`GetZeroBasedValueOffsets` returns `array.value_offsets()`
unchanged when `length() == 0`). nanoarrow's IPC writer does the same.
Apache.Arrow's `ArrowStreamReader` reads such a batch fine. Only concatenating
it fails.
Seen on Apache.Arrow 23.0.0 (.NET 10, Linux arm64). The code is unchanged on
`main` as of 117da35.
## Cause
`ConcatenateVariableBinaryValueBuffer()`
(`src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:972`) and
`ConcatenateLargeVariableBinaryValueBuffer()` (`:869`) slice `Length + 1`
offsets from every input:
```csharp
var offsets =
arrayData.Buffers[1].Span.CastTo<int>().Slice(arrayData.Offset,
arrayData.Length + 1);
```
For a length-0 input with an empty offsets buffer, that slices one element
out of an empty span and throws. The neighbouring list paths (`:758`) and the
offset-buffer concatenators `ConcatenateOffsetBuffer()` (`:995`) and
`ConcatenateLargeOffsetBuffer()` already skip such inputs with `if
(arrayData.Length == 0) continue;`. Those two functions are missing the same
guard. A length-0 input contributes no value bytes, so skipping it is correct.
## Repro (pure C#)
```csharp
using Apache.Arrow;
using Apache.Arrow.Types;
// A zero-length string array with an empty (0-byte) offsets buffer, as read
from IPC
var empty = new StringArray(new ArrayData(StringType.Default, length: 0,
nullCount: 0, offset: 0,
new[] { ArrowBuffer.Empty, ArrowBuffer.Empty, ArrowBuffer.Empty }));
var other = new StringArray.Builder().Append("a").Build();
Console.WriteLine(ArrowArrayConcatenator.Concatenate(new IArrowArray[] {
empty, other }).Length);
```
```
Unhandled exception. System.ArgumentOutOfRangeException: Specified argument
was out of the range of valid values.
at
Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.ConcatenateVariableBinaryValueBuffer()
at
Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.ConcatenateVariableBinaryArrayData(IArrowType
type)
at
Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.Visit(StringType
type)
```
## Repro (IPC from pyarrow)
This writes a stream whose `string`, `large_string` and `list<int32>`
columns have 0-byte offsets buffers. pyarrow 26.0.0 validates it with
`validate(full=True)` before writing and after reading it back:
```python
import pyarrow as pa, pyarrow.ipc as ipc
cols = {
"s": pa.Array.from_buffers(pa.string(), 0, [None, None,
pa.py_buffer(b"")]),
"ls": pa.Array.from_buffers(pa.large_string(), 0, [None, None,
pa.py_buffer(b"")]),
"l": pa.Array.from_buffers(pa.list_(pa.int32()), 0, [None, None],
children=[pa.array([], pa.int32())]),
}
batch = pa.record_batch(list(cols.values()), names=list(cols))
with ipc.new_stream("empty_offsets.arrows", batch.schema) as w:
w.write_batch(batch)
```
Reading it with `ArrowStreamReader` and calling
`ArrowArrayConcatenator.Concatenate(new[] { col, col })` on each column gives:
```
column 0 utf8 length=0
ArgumentOutOfRangeException at ...ConcatenateVariableBinaryValueBuffer()
column 1 large_utf8 length=0
ArgumentOutOfRangeException at
...ConcatenateLargeVariableBinaryValueBuffer()
column 2 list length=0
Concatenate([col, col]) length=0 <- lists already skip empty inputs
```
**Expected:** the empty input contributes nothing, and the concatenation
succeeds.
**Suggested fix:** add `if (arrayData.Length == 0) continue;` to the loops
in `ConcatenateVariableBinaryValueBuffer()` and
`ConcatenateLargeVariableBinaryValueBuffer()`, as the other offset-reading
paths in the file already do.
## Context
I hit this when a DuckDB extension started writing Arrow IPC with nanoarrow,
and a .NET worker using Apache.Arrow concatenated the incoming batches. The
value array of an empty `MAP` arrived as a zero-length `string` with an empty
offsets buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
