This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 1d1607a761 Move binary and byte-view cast helpers into a private
submodule (#11077)
1d1607a761 is described below
commit 1d1607a761e1dff1a104aefcf6ff248f2579818c
Author: Anand Hegde <[email protected]>
AuthorDate: Sat Oct 3 03:44:27 2026 +0000
Move binary and byte-view cast helpers into a private submodule (#11077)
# Which issue does this PR close?
Part of #11032 — the **binary, fixed-size binary and byte-view helpers**
group
only, so it does not close the issue.
Sibling PRs against the same issue: #11075 (temporal) and #11076
(numeric and
boolean). All three are independent and move disjoint functions, but
each adds a
`mod` line and an import to `mod.rs`, so whichever lands later will want
a
trivial rebase.
# Rationale for this change
`arrow-cast/src/cast/mod.rs` is ~14k lines. #5125 moved list, decimal,
dictionary, string, map, run-array and union support into private
submodules,
and #11032 lists the groups still inline. The byte-oriented conversions
are one
of them.
# What changes are included in this PR?
A new private `arrow-cast/src/cast/binary.rs` holding:
- `cast_numeric_to_binary`
- `cast_binary_to_fixed_size_binary`
- `cast_fixed_size_binary_to_binary`
- `cast_fixed_size_binary_to_binary_view`
- `cast_byte_container`
- `cast_view_to_byte`
`cast_with_options` stays the top-level dispatcher in `mod.rs`.
Worth flagging for review:
- **No public API surface is involved here.** Unlike the numeric group
in
#11076, none of these six were `pub`, so `mod.rs` uses a plain
`use crate::cast::binary::*` rather than a `pub use`. All six are called
from
the dispatcher, so all six became `pub(crate)` and nothing stayed
private.
- **`cast_numeric_to_binary` is not adjacent to the rest.** It sits up
with the
numeric helpers in `mod.rs` rather than with the other byte conversions,
but
it belongs to this group, so it is moved from there.
- Three imports left orphaned in `mod.rs` by the move are dropped:
`arrow_data::ByteView`, and `Buffer` from the `arrow_buffer` import
list.
# Are these changes tested?
Covered by the existing `arrow-cast` tests — this is a code move, so no
new
tests are warranted and no existing test was modified.
```
cargo test -p arrow-cast 379 passed + 12 passed, 0 failed (same as base)
cargo clippy -p arrow-cast --all-targets -- -D warnings clean
cargo fmt -p arrow-cast -- --check clean
cargo build -p arrow builds
```
As with the sibling PRs, the faithfulness of the move is checked
mechanically
rather than by eye: extracting the two original line ranges from `HEAD`
and
normalising away the added `pub(crate)` gives a **byte-identical** match
against
the new file.
Run on macOS 26.6.2, aarch64-apple-darwin, toolchain 1.98.1 from
`rust-toolchain.toml`.
# Are there any user-facing changes?
No. No public API change and no behavioural change.
# AI usage disclosure
Written with AI assistance (Claude). The AI performed the mechanical
extraction
and ran the verification above; I reviewed the result and own the
change.
The moved bodies are unedited, and the byte-identity check is the
evidence for
that rather than a reading. The judgement calls — pulling
`cast_numeric_to_binary`
out of the numeric neighbourhood, the uniform `pub(crate)`, and the
three dropped
imports — are listed above because they are the parts a "pure move" diff
would
not otherwise draw attention to.
---
arrow-cast/src/cast/binary.rs | 198 ++++++++++++++++++++++++++++++++++++++++++
arrow-cast/src/cast/mod.rs | 176 +------------------------------------
2 files changed, 201 insertions(+), 173 deletions(-)
diff --git a/arrow-cast/src/cast/binary.rs b/arrow-cast/src/cast/binary.rs
new file mode 100644
index 0000000000..72ce5ae1a9
--- /dev/null
+++ b/arrow-cast/src/cast/binary.rs
@@ -0,0 +1,198 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Cast support for binary, fixed-size binary and byte-view arrays.
+
+use arrow_array::{builder::*, cast::*, types::*, *};
+use arrow_buffer::{Buffer, OffsetBuffer};
+use arrow_data::{ArrayData, ByteView};
+use arrow_schema::{ArrowError, DataType};
+use num_traits::{NumCast, ToPrimitive};
+use std::sync::Arc;
+
+use super::CastOptions;
+
+pub(crate) fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O:
OffsetSizeTrait>(
+ array: &dyn Array,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array.as_primitive::<FROM>();
+ let size = std::mem::size_of::<FROM::Native>();
+ let offsets = OffsetBuffer::from_repeated_length(size, array.len());
+ Ok(Arc::new(GenericBinaryArray::<O>::try_new(
+ offsets,
+ array.values().inner().clone(),
+ array.nulls().cloned(),
+ )?))
+}
+
+/// Helper function to cast from one `BinaryArray` or 'LargeBinaryArray' to
'FixedSizeBinaryArray'.
+pub(crate) fn cast_binary_to_fixed_size_binary<O: OffsetSizeTrait>(
+ array: &dyn Array,
+ byte_width: i32,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array.as_binary::<O>();
+ let mut builder = FixedSizeBinaryBuilder::with_capacity(array.len(),
byte_width);
+
+ for i in 0..array.len() {
+ if array.is_null(i) {
+ builder.append_null();
+ } else {
+ match builder.append_value(array.value(i)) {
+ Ok(()) => {}
+ Err(e) => match cast_options.safe {
+ true => builder.append_null(),
+ false => return Err(e),
+ },
+ }
+ }
+ }
+
+ Ok(Arc::new(builder.finish()))
+}
+
+/// Helper function to cast from 'FixedSizeBinaryArray' to one `BinaryArray`
or 'LargeBinaryArray'.
+/// If the target one is too large for the source array it will return an
Error.
+pub(crate) fn cast_fixed_size_binary_to_binary<O: OffsetSizeTrait>(
+ array: &dyn Array,
+ byte_width: i32,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array
+ .as_any()
+ .downcast_ref::<FixedSizeBinaryArray>()
+ .unwrap();
+
+ let offsets: i128 = byte_width as i128 * array.len() as i128;
+
+ let is_binary = matches!(GenericBinaryType::<O>::DATA_TYPE,
DataType::Binary);
+ if is_binary && offsets > i32::MAX as i128 {
+ return Err(ArrowError::ComputeError(
+ "FixedSizeBinary array too large to cast to Binary
array".to_string(),
+ ));
+ } else if !is_binary && offsets > i64::MAX as i128 {
+ return Err(ArrowError::ComputeError(
+ "FixedSizeBinary array too large to cast to LargeBinary
array".to_string(),
+ ));
+ }
+
+ let mut builder = GenericBinaryBuilder::<O>::with_capacity(array.len(),
array.len());
+
+ for i in 0..array.len() {
+ if array.is_null(i) {
+ builder.append_null();
+ } else {
+ builder.append_value(array.value(i));
+ }
+ }
+
+ Ok(Arc::new(builder.finish()))
+}
+
+pub(crate) fn cast_fixed_size_binary_to_binary_view(
+ array: &dyn Array,
+ _byte_width: i32,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array
+ .as_any()
+ .downcast_ref::<FixedSizeBinaryArray>()
+ .unwrap();
+
+ let mut builder = BinaryViewBuilder::with_capacity(array.len());
+ for i in 0..array.len() {
+ if array.is_null(i) {
+ builder.append_null();
+ } else {
+ builder.append_value(array.value(i));
+ }
+ }
+
+ Ok(Arc::new(builder.finish()))
+}
+
+/// Helper function to cast from one `ByteArrayType` to another and vice versa.
+/// If the target one (e.g., `LargeUtf8`) is too large for the source array it
will return an Error.
+pub(crate) fn cast_byte_container<FROM, TO>(array: &dyn Array) ->
Result<ArrayRef, ArrowError>
+where
+ FROM: ByteArrayType,
+ TO: ByteArrayType<Native = FROM::Native>,
+ FROM::Offset: OffsetSizeTrait + ToPrimitive,
+ TO::Offset: OffsetSizeTrait + NumCast,
+{
+ let data = array.to_data();
+ assert_eq!(data.data_type(), &FROM::DATA_TYPE);
+ let str_values_buf = data.buffers()[1].clone();
+ let offsets = data.buffers()[0].typed_data::<FROM::Offset>();
+
+ let mut cast_offsets = Vec::<TO::Offset>::with_capacity(offsets.len());
+ offsets
+ .iter()
+ .try_for_each::<_, Result<_, ArrowError>>(|offset| {
+ let offset =
+ <<TO as ByteArrayType>::Offset as
NumCast>::from(*offset).ok_or_else(|| {
+ ArrowError::ComputeError(format!(
+ "{}{} array too large to cast to {}{} array",
+ FROM::Offset::PREFIX,
+ FROM::PREFIX,
+ TO::Offset::PREFIX,
+ TO::PREFIX
+ ))
+ })?;
+ cast_offsets.push(offset);
+ Ok(())
+ })?;
+
+ let offset_buffer = Buffer::from_vec(cast_offsets);
+
+ let dtype = TO::DATA_TYPE;
+
+ let builder = ArrayData::builder(dtype)
+ .offset(array.offset())
+ .len(array.len())
+ .add_buffer(offset_buffer)
+ .add_buffer(str_values_buf)
+ .nulls(data.nulls().cloned());
+
+ let array_data = unsafe { builder.build_unchecked() };
+
+ Ok(Arc::new(GenericByteArray::<TO>::from(array_data)))
+}
+
+/// Helper function to cast from one `ByteViewType` array to `ByteArrayType`
array.
+pub(crate) fn cast_view_to_byte<FROM, TO>(array: &dyn Array) ->
Result<ArrayRef, ArrowError>
+where
+ FROM: ByteViewType,
+ TO: ByteArrayType,
+ FROM::Native: AsRef<TO::Native>,
+{
+ let data = array.to_data();
+ let view_array = GenericByteViewArray::<FROM>::from(data);
+
+ let len = view_array.len();
+ let bytes = view_array
+ .views()
+ .iter()
+ .map(|v| ByteView::from(*v).length as usize)
+ .sum::<usize>();
+
+ let mut byte_array_builder = GenericByteBuilder::<TO>::with_capacity(len,
bytes);
+
+ for val in &view_array {
+ byte_array_builder.append_option(val);
+ }
+
+ Ok(Arc::new(byte_array_builder.finish()))
+}
diff --git a/arrow-cast/src/cast/mod.rs b/arrow-cast/src/cast/mod.rs
index 4bff879292..0887345d05 100644
--- a/arrow-cast/src/cast/mod.rs
+++ b/arrow-cast/src/cast/mod.rs
@@ -37,6 +37,7 @@
//! assert_eq!(7.0, c.value(2));
//! ```
+mod binary;
mod decimal;
mod dictionary;
mod list;
@@ -46,6 +47,7 @@ mod string;
mod structs;
mod union;
+use crate::cast::binary::*;
use crate::cast::decimal::*;
use crate::cast::dictionary::*;
use crate::cast::list::*;
@@ -56,7 +58,6 @@ use crate::cast::structs::*;
pub use crate::cast::union::*;
use arrow_buffer::IntervalMonthDayNano;
-use arrow_data::ByteView;
use chrono::{
FixedOffset, LocalResult, NaiveDateTime, NaiveTime, Offset, TimeDelta,
TimeZone, Utc,
};
@@ -69,7 +70,7 @@ use crate::parse::{
string_to_datetime,
};
use arrow_array::{builder::*, cast::*, temporal_conversions::*, timezone::Tz,
types::*, *};
-use arrow_buffer::{ArrowNativeType, Buffer, OffsetBuffer, i256};
+use arrow_buffer::{ArrowNativeType, OffsetBuffer, i256};
use arrow_data::ArrayData;
use arrow_data::transform::MutableArrayData;
use arrow_schema::*;
@@ -2632,19 +2633,6 @@ where
num_traits::cast::cast::<I, O>(value)
}
-fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O: OffsetSizeTrait>(
- array: &dyn Array,
-) -> Result<ArrayRef, ArrowError> {
- let array = array.as_primitive::<FROM>();
- let size = std::mem::size_of::<FROM::Native>();
- let offsets = OffsetBuffer::from_repeated_length(size, array.len());
- Ok(Arc::new(GenericBinaryArray::<O>::try_new(
- offsets,
- array.values().inner().clone(),
- array.nulls().cloned(),
- )?))
-}
-
/// Returns the offset to use when interpreting `local` as a wall clock reading
/// in `tz`, or `None` if it cannot be resolved.
///
@@ -2801,164 +2789,6 @@ where
}
}
-/// Helper function to cast from one `BinaryArray` or 'LargeBinaryArray' to
'FixedSizeBinaryArray'.
-fn cast_binary_to_fixed_size_binary<O: OffsetSizeTrait>(
- array: &dyn Array,
- byte_width: i32,
- cast_options: &CastOptions,
-) -> Result<ArrayRef, ArrowError> {
- let array = array.as_binary::<O>();
- let mut builder = FixedSizeBinaryBuilder::with_capacity(array.len(),
byte_width);
-
- for i in 0..array.len() {
- if array.is_null(i) {
- builder.append_null();
- } else {
- match builder.append_value(array.value(i)) {
- Ok(()) => {}
- Err(e) => match cast_options.safe {
- true => builder.append_null(),
- false => return Err(e),
- },
- }
- }
- }
-
- Ok(Arc::new(builder.finish()))
-}
-
-/// Helper function to cast from 'FixedSizeBinaryArray' to one `BinaryArray`
or 'LargeBinaryArray'.
-/// If the target one is too large for the source array it will return an
Error.
-fn cast_fixed_size_binary_to_binary<O: OffsetSizeTrait>(
- array: &dyn Array,
- byte_width: i32,
-) -> Result<ArrayRef, ArrowError> {
- let array = array
- .as_any()
- .downcast_ref::<FixedSizeBinaryArray>()
- .unwrap();
-
- let offsets: i128 = byte_width as i128 * array.len() as i128;
-
- let is_binary = matches!(GenericBinaryType::<O>::DATA_TYPE,
DataType::Binary);
- if is_binary && offsets > i32::MAX as i128 {
- return Err(ArrowError::ComputeError(
- "FixedSizeBinary array too large to cast to Binary
array".to_string(),
- ));
- } else if !is_binary && offsets > i64::MAX as i128 {
- return Err(ArrowError::ComputeError(
- "FixedSizeBinary array too large to cast to LargeBinary
array".to_string(),
- ));
- }
-
- let mut builder = GenericBinaryBuilder::<O>::with_capacity(array.len(),
array.len());
-
- for i in 0..array.len() {
- if array.is_null(i) {
- builder.append_null();
- } else {
- builder.append_value(array.value(i));
- }
- }
-
- Ok(Arc::new(builder.finish()))
-}
-
-fn cast_fixed_size_binary_to_binary_view(
- array: &dyn Array,
- _byte_width: i32,
-) -> Result<ArrayRef, ArrowError> {
- let array = array
- .as_any()
- .downcast_ref::<FixedSizeBinaryArray>()
- .unwrap();
-
- let mut builder = BinaryViewBuilder::with_capacity(array.len());
- for i in 0..array.len() {
- if array.is_null(i) {
- builder.append_null();
- } else {
- builder.append_value(array.value(i));
- }
- }
-
- Ok(Arc::new(builder.finish()))
-}
-
-/// Helper function to cast from one `ByteArrayType` to another and vice versa.
-/// If the target one (e.g., `LargeUtf8`) is too large for the source array it
will return an Error.
-fn cast_byte_container<FROM, TO>(array: &dyn Array) -> Result<ArrayRef,
ArrowError>
-where
- FROM: ByteArrayType,
- TO: ByteArrayType<Native = FROM::Native>,
- FROM::Offset: OffsetSizeTrait + ToPrimitive,
- TO::Offset: OffsetSizeTrait + NumCast,
-{
- let data = array.to_data();
- assert_eq!(data.data_type(), &FROM::DATA_TYPE);
- let str_values_buf = data.buffers()[1].clone();
- let offsets = data.buffers()[0].typed_data::<FROM::Offset>();
-
- let mut cast_offsets = Vec::<TO::Offset>::with_capacity(offsets.len());
- offsets
- .iter()
- .try_for_each::<_, Result<_, ArrowError>>(|offset| {
- let offset =
- <<TO as ByteArrayType>::Offset as
NumCast>::from(*offset).ok_or_else(|| {
- ArrowError::ComputeError(format!(
- "{}{} array too large to cast to {}{} array",
- FROM::Offset::PREFIX,
- FROM::PREFIX,
- TO::Offset::PREFIX,
- TO::PREFIX
- ))
- })?;
- cast_offsets.push(offset);
- Ok(())
- })?;
-
- let offset_buffer = Buffer::from_vec(cast_offsets);
-
- let dtype = TO::DATA_TYPE;
-
- let builder = ArrayData::builder(dtype)
- .offset(array.offset())
- .len(array.len())
- .add_buffer(offset_buffer)
- .add_buffer(str_values_buf)
- .nulls(data.nulls().cloned());
-
- let array_data = unsafe { builder.build_unchecked() };
-
- Ok(Arc::new(GenericByteArray::<TO>::from(array_data)))
-}
-
-/// Helper function to cast from one `ByteViewType` array to `ByteArrayType`
array.
-fn cast_view_to_byte<FROM, TO>(array: &dyn Array) -> Result<ArrayRef,
ArrowError>
-where
- FROM: ByteViewType,
- TO: ByteArrayType,
- FROM::Native: AsRef<TO::Native>,
-{
- let data = array.to_data();
- let view_array = GenericByteViewArray::<FROM>::from(data);
-
- let len = view_array.len();
- let bytes = view_array
- .views()
- .iter()
- .map(|v| ByteView::from(*v).length as usize)
- .sum::<usize>();
-
- let mut byte_array_builder = GenericByteBuilder::<TO>::with_capacity(len,
bytes);
-
- for val in &view_array {
- byte_array_builder.append_option(val);
- }
-
- Ok(Arc::new(byte_array_builder.finish()))
-}
-
#[cfg(test)]
mod tests {
use super::*;
