mbutrovich commented on code in PR #11710:
URL: https://github.com/apache/arrow-rs/pull/11710#discussion_r4339284213
##########
arrow-array/src/array/run_array.rs:
##########
@@ -73,12 +73,62 @@ macro_rules! ree_map {
/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘ │ B │
/// ├─────────────────┤
/// RunArray │ B │ run length
of 'B' = run_ends[2] - run_ends[1] = 3
-/// length = 3 └─────────────────┘
+/// offset = 0, len = 6 └─────────────────┘
///
/// Logical array
/// Contents
/// ```
///
+/// # `offset` and `len`
+///
+/// The [`Array::len`] and [`Array::offset`] for a `RunArray` refer to the
+/// **logical** length, not the `run_ends` and `values` child arrays. In the
+/// example above the `RunArray` has length `6` even though `run_ends` and
+/// `values` each contain `3` elements.
+///
+/// The values in `run_ends` represent indexes of the *unsliced* logical array.
+/// This allows zero-copy [`RunArray::slice`]: `run_ends` and `values` are left
+/// untouched and only `offset` and `len` are adjusted.
+///
+/// A [`RunArray`] must satisfy the following invariants:
+///
+/// * every run end is greater than zero and `run_ends` is strictly increasing
+/// * `run_ends` has no null values
+/// * `run_ends` and `values` have the same length
+/// * if `len > 0`, the last run end is greater than or equal to `offset +
len`.
Review Comment:
> if `len > 0`, the last run end is greater than or equal to `offset + len`.
Should the docs say that this condition comes from arrow-rs and not from the
spec, and that C++ is stricter?
The [run-end encoded (REE)
layout](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/docs/source/format/Columnar.rst?plain=1#L1089-L1160)
defines run ends as the accumulated length of the runs. It does not mention a
parent offset or an empty array. The C Data Interface defines
[`offset`](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/docs/source/format/CDataInterface.rst?plain=1#L456-L464)
as the logical offset inside the array, and says buffers must hold [`length +
offset`
values](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/docs/source/format/CDataInterface.rst?plain=1#L480-L485).
REE has no buffers, so as I read it the spec leaves `len == 0` open.
C++ has no `len > 0` condition. It requires the last run end to be at least
`offset + length` [whenever the run ends child is
non-empty](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/cpp/src/arrow/util/ree_util.cc#L247-L254),
and exempts `length == 0` only [when the run ends child is
empty](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/cpp/src/arrow/util/ree_util.cc#L233-L239).
So with run ends `[1]`, `offset = 2` and `len = 0` passes
`ArrayData::validate_full` on `main` and, as I read the C++ code, fails C++
validation. I have not run the C++ side. The arrow-rs side is
[`check_run_ends`](https://github.com/apache/arrow-rs/blob/322f43c2bfeb7b98ec4264c2026e3fc683486127/arrow-data/src/data.rs#L1878)
and
[`RunEndBuffer::new`](https://github.com/apache/arrow-rs/blob/322f43c2bfeb7b98ec4264c2026e3fc683486127/arrow-buffer/src/buffer/run.rs#L124).
I could not find where the `len == 0` exemption was decided. Is it intended?
If so, a sentence here noting that C++ would reject such an array could help
anyone exporting one over FFI or IPC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
