mbutrovich commented on code in PR #12228:
URL: https://github.com/apache/arrow-rs/pull/12228#discussion_r4589508471
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -201,33 +202,44 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
/// Builds the [`GenericByteArray`] and reset this builder.
pub fn finish(&mut self) -> GenericByteArray<T> {
- let array_type = T::DATA_TYPE;
- let array_builder = ArrayDataBuilder::new(array_type)
- .len(self.len())
- .add_buffer(std::mem::take(&mut self.offsets_builder).into())
- .add_buffer(std::mem::take(&mut self.value_builder).into())
- .nulls(self.null_buffer_builder.finish());
+ // SAFETY: offsets start at 0 and every append pushes the current end
of
+ // `value_builder`, so they are non-empty and monotonically increasing.
Review Comment:
Could we add a `debug_assert!` for this `SAFETY` claim, here and at lines
226-227 in `finish_cloned`?
The old `finish` went through `ArrayDataBuilder::build_unchecked`, which
runs full validation under `force_validate`, including the check that offsets
never decrease
([data.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-data/src/data.rs#L1776-L1790)).
The new path loses that check. `OffsetBuffer::new_unchecked` has no
`force_validate` branch
([offset.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-buffer/src/buffer/offset.rs#L68-L76)),
and `GenericByteArray::try_new` assumes the offsets are already monotonic
([types.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-array/src/types.rs#L1732-L1741)).
With `--features force_validate`, `BinaryArray::new_unchecked` and
`StringArray::new_unchecked` both accept the offsets `[0, 2, 1, 3]`.
Could you also update this sentence in the PR description? It holds for the
other three builders, but not for offset ordering here. Every other caller of
`OffsetBuffer::new_unchecked` has the same gap, which is #12284, so the assert
covers only this builder until that's fixed.
> The existing `force_validate` feature still routes these through
validation.
```suggestion
debug_assert!(self.offsets_builder.is_sorted());
// SAFETY: offsets start at 0 and every append pushes the current
end of
// `value_builder`, so they are non-empty and monotonically
increasing.
```
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -201,33 +202,44 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
/// Builds the [`GenericByteArray`] and reset this builder.
pub fn finish(&mut self) -> GenericByteArray<T> {
- let array_type = T::DATA_TYPE;
- let array_builder = ArrayDataBuilder::new(array_type)
- .len(self.len())
- .add_buffer(std::mem::take(&mut self.offsets_builder).into())
- .add_buffer(std::mem::take(&mut self.value_builder).into())
- .nulls(self.null_buffer_builder.finish());
+ // SAFETY: offsets start at 0 and every append pushes the current end
of
+ // `value_builder`, so they are non-empty and monotonically increasing.
+ let offsets = unsafe {
+ OffsetBuffer::new_unchecked(std::mem::take(&mut
self.offsets_builder).into())
+ };
+ let values = std::mem::take(&mut self.value_builder).into();
+ let nulls = self
+ .null_buffer_builder
+ .finish()
+ .filter(|n| n.null_count() > 0);
- self.offsets_builder.push(self.next_offset());
- // SAFETY: builder is constructed from valid offset and value buffers
maintained by the builder
- let array_data = unsafe { array_builder.build_unchecked() };
- GenericByteArray::from(array_data)
+ self.offsets_builder.push(T::Offset::usize_as(0));
+
+ // SAFETY: `try_new` would not error. The last offset is
`values.len()`,
+ // `nulls` has one entry per value, and for strings every value was
+ // appended from a `&str`, so each offset pair bounds valid UTF-8.
Review Comment:
I got part of the text I suggested for this comment wrong. The last offset
can be less than `values.len()`, because `write!` adds bytes to `value_builder`
without pushing an offset
([`write_str`](https://github.com/apache/arrow-rs/blob/c3370f73078ca0d18167554504784ef2d0ed97e2/arrow-array/src/builder/generic_bytes_builder.rs#L390-L395),
and `std::io::Write::write` on `GenericBinaryBuilder` does the same). After
`append_value("a")` and then `write!(builder, "foo")`, `finish` returns an
array whose last offset is 1 and whose values are 4 bytes long. `try_new` only
requires the last offset to be at most `values.len()`
([types.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-array/src/types.rs#L1683-L1691)),
so the code is sound, but the comment should say that. Could we use this text
here and at lines 239-241 in `finish_cloned`?
```suggestion
// SAFETY: `try_new` would not error. No append pushes an offset past
// the end of `value_builder`, which only grows, so the last offset
is
// at most `values.len()` (`write!` can leave bytes after it).
`nulls`
// has one entry per value. For strings, every append, `write!`
// included, copies whole UTF-8 sequences, so each offset pair bounds
// valid UTF-8.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]