Awesome06 opened a new pull request, #55171:
URL: https://github.com/apache/arrow/pull/55171
### Rationale for this change
Currently, Arrow compute supports comparing `Utf8` with `LargeUtf8` and
comparing `Utf8View` with `Utf8View`, but attempting to compare a `Utf8View`
array/scalar with either `Utf8` or `LargeUtf8` fails with:
```text
ArrowNotImplementedError: Function 'equal' has no kernel matching input
types (string, string_view)
```
In `scalar_compare.cc`, comparison operators (`equal`, `not_equal`, `less`,
`greater`, etc.) enforce homogeneous symmetric kernel signatures (`{ty, ty}`)
and rely on `DispatchBest` to resolve heterogeneous types via `CommonBinary()`.
However, `CommonBinary()` did not previously recognize `Type::STRING_VIEW` or
`Type::BINARY_VIEW`, returning `nullptr` and causing `DispatchBest` to fail.
When mixing string/binary view types with offset-based string/binary types,
promoting to the view type (`utf8_view` or `binary_view`) is optimal:
1. `ZeroCopyCastExec` in `scalar_cast_string.cc` reuses the existing
character data buffer directly as variadic buffer 0, only allocating 16-byte
descriptors without copying string payload data. (Conversely, promoting to
`LargeUtf8` would require a deep copy of all string data into contiguous
buffers).
2. For scalar-array comparisons (`string_view_array == "literal"`), only the
scalar literal is converted (O(1) zero-allocation), leaving array memory
completely untouched.
### What changes are included in this PR?
1. `cpp/src/arrow/compute/kernels/codegen_internal.cc`:
- Extended `CommonBinary()` to handle `Type::STRING_VIEW` and
`Type::BINARY_VIEW`.
- Promotes mixed string types to `utf8_view()`, and mixed
binary/fixed-size types to `binary_view()`.
2. `cpp/src/arrow/compute/kernels/scalar_compare_test.cc`:
- Added `CheckDispatchBest` assertions for view <-> non-view binary and
string combinations (`utf8`, `large_utf8`, `utf8_view`, `binary`,
`large_binary`, `binary_view`, `fixed_size_binary`).
- Added `TEST(TestBinaryViewCompareKernel, MixedStringTypes)` validating:
- Short strings (<= 12 bytes, testing inline descriptors)
- Long strings (> 12 bytes, testing out-of-line prefix + buffer lookup)
- Sliced arrays (`array.Slice(1, 2)`)
- Empty arrays (`[]`) and all-null arrays (`[null, null]`)
- Array-scalar and scalar-array operations across `equal`, `not_equal`,
`less`, and `greater`.
3. `python/pyarrow/tests/test_compute.py`:
- Added `test_compare_mixed_string_types` parameterized across all 9
permutations of `string`, `large_string`, and `string_view` (including sliced
and >12-byte strings).
- Added `test_compare_mixed_binary_types` parameterized across all 9
permutations of `binary`, `large_binary`, and `binary_view`.
### Are these changes tested?
Yes:
- C++ unit tests in `arrow-compute-scalar-math-test` (including
`TestCompareKernel.DispatchBest` and
`TestBinaryViewCompareKernel.MixedStringTypes`). All 594 tests in the test
suite pass with 0 failures.
- PyArrow unit tests added to `python/pyarrow/tests/test_compute.py`.
### Are there any user-facing changes?
Yes. Users can now directly compare `Utf8View` / `BinaryView` columns
against `Utf8`, `LargeUtf8`, `Binary`, and `LargeBinary` columns or scalars in
expressions and compute kernels without explicit casting or
`ArrowNotImplementedError`.
### Was AI used for this PR?
AI was used to diagnose the type promotion dispatch path, draft
implementation code, and construct test cases under developer direction. The
changes and tests were reviewed, compiled, and verified locally.
**PR code and description written by:**
- [ ] Human
- [X] AI
**Reviewed before submission by:**
- [X] Human
- [X] AI
- [ ] Not reviewed
* GitHub Issue: #54668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
