mbutrovich commented on code in PR #12088:
URL: https://github.com/apache/arrow-rs/pull/12088#discussion_r4485814376
##########
arrow-array/src/array/mod.rs:
##########
@@ -432,6 +432,18 @@ pub unsafe trait Array: std::fmt::Debug + Send + Sync {
/// A reference-counted reference to a generic `Array`
pub type ArrayRef = Arc<dyn Array>;
+/// Returns `true` if two child value arrays have the same layout and backing
buffers.
+///
+/// This is useful for recognizing shared values in [`DictionaryArray`] and
[`RunArray`].
+/// It first checks whether the [`ArrayRef`]s point to the same allocation,
then falls back
+/// to [`ArrayData::ptr_eq`] to recognize shallow clones such as
`Arc::new(array.clone())`.
+///
+/// This checks storage identity, not logical equality: arrays with equal
values but
+/// different buffers return `false`.
Review Comment:
What should this return for arrays that have no allocated storage? At the
head commit, two independently built `Int32Array::from(Vec::<i32>::new())`
arrays return `true`, and so do two independent `NullArray::new(3)` arrays.
That contradicts this sentence. Returning `true` is correct for reusing a
dictionary mapping, so could we document it and add both cases to the tests?
```suggestion
/// This checks storage identity, not logical equality. Arrays with equal
values in
/// different buffers return `false`. Arrays with no allocated storage, such
as two empty
/// primitive arrays built from empty `Vec`s or two [`NullArray`]s of the
same length,
/// return `true`.
```
##########
arrow-array/src/array/mod.rs:
##########
@@ -432,6 +432,18 @@ pub unsafe trait Array: std::fmt::Debug + Send + Sync {
/// A reference-counted reference to a generic `Array`
pub type ArrayRef = Arc<dyn Array>;
+/// Returns `true` if two child value arrays have the same layout and backing
buffers.
+///
+/// This is useful for recognizing shared values in [`DictionaryArray`] and
[`RunArray`].
+/// It first checks whether the [`ArrayRef`]s point to the same allocation,
then falls back
+/// to [`ArrayData::ptr_eq`] to recognize shallow clones such as
`Arc::new(array.clone())`.
+///
+/// This checks storage identity, not logical equality: arrays with equal
values but
+/// different buffers return `false`.
+pub fn child_values_ptr_eq(left: &ArrayRef, right: &ArrayRef) -> bool {
Review Comment:
Do we expect callers outside arrow-rs to need this? If not, could it be a
`pub(crate)` function in `arrow-select/src/dictionary.rs` for now? The caller
planned in #11743 is in `arrow-select`, and
[`should_merge_dictionary_values`](https://github.com/apache/arrow-rs/blob/cc58320c2da359b757814db5a9651397d829b601/arrow-select/src/dictionary.rs#L137-L196)
already makes the same check there. A `pub(crate)` function in `arrow-array`
wouldn't be visible to `arrow-select`, so keeping it crate-private means moving
it. A public function is permanent once released, while a crate-private one can
become public later if a downstream use comes up.
The existing check also avoids `to_data()` for byte arrays. For a
`StringArray`, `to_data()` clones the array and [builds a new `Vec` of
buffers](https://github.com/apache/arrow-rs/blob/cc58320c2da359b757814db5a9651397d829b601/arrow-array/src/array/byte_array.rs#L557-L567),
so each call here allocates a `Vec` for each side before comparing pointers.
`bytes_ptr_eq` compares the offsets, values, and null buffers directly.
One way to do it is to move the `PtrEq` dispatch out of
`should_merge_dictionary_values` into a `pub(crate)` function that checks
`Arc::ptr_eq` first, uses `bytes_ptr_eq` for the byte types, and falls back to
`ArrayData::ptr_eq` otherwise. `should_merge_dictionary_values` and #11743 can
then both call it, which also gives this PR a caller. If it stays public, could
it get a doctest like `new_empty_array` and `new_null_array` have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
