hanke580 opened a new issue, #54570:
URL: https://github.com/apache/arrow/issues/54570
### Describe the bug, including details regarding any error messages,
version, and platform.
### Describe the bug, including details regarding any error messages,
version, and platform.
Casting a `list`/`large_list`/`list_view` array to `fixed_size_list` crashes
the process with a segmentation fault when the array's null slots do not have
the target length and the child (values) array is empty with no data buffer. A
null list read back from IPC is enough to trigger it:
**Reproducer**
```python
import pyarrow as pa
import pyarrow.ipc as ipc
t = pa.table({"c": pa.array([None], pa.list_(pa.int64()))})
sink = pa.BufferOutputStream()
with ipc.new_stream(sink, t.schema) as w:
w.write_table(t)
col = ipc.open_stream(sink.getvalue()).read_all()["c"]
col.validate(full=True) # passes
print(col.cast(pa.list_(pa.int64(), 3))) # expected [null]
```
Output on main:
```
Segmentation fault (core dumped)
```
The IPC file format crashes too, and so do arrays built with `from_buffers`
whose zero-length child has a null data buffer (spec-valid for a zero-length
array):
```python
child = pa.Array.from_buffers(pa.int64(), 0, [None, None])
arr = pa.ListArray.from_buffers(pa.list_(pa.int64()), 1,
[pa.py_buffer(b"\x00"), pa.array([0, 0],
pa.int32()).buffers()[1]],
children=[child])
arr.cast(pa.list_(pa.int64(), 3)) # segfault
```
| input | cast to `fixed_size_list<…>[3]` |
|---|---|
| `list`, `large_list` or `list_view` with one null slot and an empty child
without a data buffer (from IPC or `from_buffers`) | segfault |
| any value type tried (`int64`, `timestamp[us]`) | segfault |
| the same array from `pa.array([None], …)` (the child gets an allocated,
zero-length data buffer) | `[None]`, but index 0 is still read out of bounds |
| a null slot of the wrong length with a non-empty child, e.g. `[[1, 2, 3],
None, [4, 5, 6]]` | correct |
We first hit this on a Polars-produced `large_list<timestamp[us]>` (Polars
`to_arrow()`): a chunk holding a single null list with an empty child.
**Backtrace** (pyarrow 25.0.1, gdb):
```
#0 arrow::compute::internal::(anonymous)::FixedWidthTakeImpl<unsigned long,
std::integral_constant<int, 64>, ...>::Exec(...)
#1 arrow::compute::internal::FixedWidthTakeExec(...)
...
#11 arrow::compute::Take(arrow::Datum const&, arrow::Datum const&,
arrow::compute::TakeOptions const&, arrow::compute::ExecContext*)
#12
arrow::compute::internal::(anonymous)::CastVarToFixedList<arrow::LargeListType>::Exec(...)
```
**Versions:** pyarrow 27.0.0.dev13 (nightly built from main at 9f03413),
Python 3.12, Linux x86_64 (Ubuntu 22.04, glibc 2.35). The released wheels
15.0.2, 17.0.0, 19.0.1, 21.0.0, 23.0.0, 25.0.1 and 26.0.0 crash the same way.
**Where it comes from**
- When a null slot does not have length `list_size`,
`CastVarToFixedList::Exec` rebuilds the values with `Take`. `BuildTakeIndices`
emits the valid index `0` for every element of a null slot ("just fill in the
null slots with first value",
[scalar_cast_nested.cc#L230-L251](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc#L230-L251)).
- `Take` is then called with `TakeOptions::NoBoundsCheck()`
([scalar_cast_nested.cc#L279-L286](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc#L279-L286)).
If the child is empty, index 0 is out of bounds. With a null data buffer,
`FixedWidthTakeImpl` dereferences a null pointer. With an allocated buffer, it
reads padding, and the result is hidden because the slot is null.
- `pc.take` on the same empty child with null indices (`[None, None, None]`)
works, so the bad indices come from the cast kernel.
- The kernel is registered for `list`, `large_list`, `list_view`,
`large_list_view` and `map` sources
([scalar_cast_nested.cc#L516-L521](https://github.com/apache/arrow/blob/9f0341314ca96eb58081d4132c77b460d880b2b0/cpp/src/arrow/compute/kernels/scalar_cast_nested.cc#L516-L521)).
Possible fixes:
- Emit a null index for the elements of a null slot (append nulls in
`BuildTakeIndices`) instead of `0`, so `Take` never reads the child for them.
- Or, when the cast child has length 0, build the output values with
`MakeArrayOfNulls(child_type, length * list_size)` instead of calling `Take`.
**Impact:** any process that casts list columns to `fixed_size_list` (for
example to build embedding or tensor columns) crashes on input read from IPC or
Arrow Flight, or received over the C Data Interface from another library, as
soon as one batch holds only null lists. The crash cannot be caught from Python.
### Component(s)
C++, Python
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
