This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-11077-7c88505f9e3bb5b5275aa2838e7ef0757962964b in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
commit 1d1607a761e1dff1a104aefcf6ff248f2579818c Author: Anand Hegde <[email protected]> AuthorDate: Sat Oct 3 03:44:27 2026 +0000 Move binary and byte-view cast helpers into a private submodule (#11077) # Which issue does this PR close? Part of #11032 — the **binary, fixed-size binary and byte-view helpers** group only, so it does not close the issue. Sibling PRs against the same issue: #11075 (temporal) and #11076 (numeric and boolean). All three are independent and move disjoint functions, but each adds a `mod` line and an import to `mod.rs`, so whichever lands later will want a trivial rebase. # Rationale for this change `arrow-cast/src/cast/mod.rs` is ~14k lines. #5125 moved list, decimal, dictionary, string, map, run-array and union support into private submodules, and #11032 lists the groups still inline. The byte-oriented conversions are one of them. # What changes are included in this PR? A new private `arrow-cast/src/cast/binary.rs` holding: - `cast_numeric_to_binary` - `cast_binary_to_fixed_size_binary` - `cast_fixed_size_binary_to_binary` - `cast_fixed_size_binary_to_binary_view` - `cast_byte_container` - `cast_view_to_byte` `cast_with_options` stays the top-level dispatcher in `mod.rs`. Worth flagging for review: - **No public API surface is involved here.** Unlike the numeric group in #11076, none of these six were `pub`, so `mod.rs` uses a plain `use crate::cast::binary::*` rather than a `pub use`. All six are called from the dispatcher, so all six became `pub(crate)` and nothing stayed private. - **`cast_numeric_to_binary` is not adjacent to the rest.** It sits up with the numeric helpers in `mod.rs` rather than with the other byte conversions, but it belongs to this group, so it is moved from there. - Three imports left orphaned in `mod.rs` by the move are dropped: `arrow_data::ByteView`, and `Buffer` from the `arrow_buffer` import list. # Are these changes tested? Covered by the existing `arrow-cast` tests — this is a code move, so no new tests are warranted and no existing test was modified. ``` cargo test -p arrow-cast 379 passed + 12 passed, 0 failed (same as base) cargo clippy -p arrow-cast --all-targets -- -D warnings clean cargo fmt -p arrow-cast -- --check clean cargo build -p arrow builds ``` As with the sibling PRs, the faithfulness of the move is checked mechanically rather than by eye: extracting the two original line ranges from `HEAD` and normalising away the added `pub(crate)` gives a **byte-identical** match against the new file. Run on macOS 26.6.2, aarch64-apple-darwin, toolchain 1.98.1 from `rust-toolchain.toml`. # Are there any user-facing changes? No. No public API change and no behavioural change. # AI usage disclosure Written with AI assistance (Claude). The AI performed the mechanical extraction and ran the verification above; I reviewed the result and own the change. The moved bodies are unedited, and the byte-identity check is the evidence for that rather than a reading. The judgement calls — pulling `cast_numeric_to_binary` out of the numeric neighbourhood, the uniform `pub(crate)`, and the three dropped imports — are listed above because they are the parts a "pure move" diff would not otherwise draw attention to. --- arrow-cast/src/cast/binary.rs | 198 ++++++++++++++++++++++++++++++++++++++++++ arrow-cast/src/cast/mod.rs | 176 +------------------------------------ 2 files changed, 201 insertions(+), 173 deletions(-) diff --git a/arrow-cast/src/cast/binary.rs b/arrow-cast/src/cast/binary.rs new file mode 100644 index 0000000000..72ce5ae1a9 --- /dev/null +++ b/arrow-cast/src/cast/binary.rs @@ -0,0 +1,198 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Cast support for binary, fixed-size binary and byte-view arrays. + +use arrow_array::{builder::*, cast::*, types::*, *}; +use arrow_buffer::{Buffer, OffsetBuffer}; +use arrow_data::{ArrayData, ByteView}; +use arrow_schema::{ArrowError, DataType}; +use num_traits::{NumCast, ToPrimitive}; +use std::sync::Arc; + +use super::CastOptions; + +pub(crate) fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O: OffsetSizeTrait>( + array: &dyn Array, +) -> Result<ArrayRef, ArrowError> { + let array = array.as_primitive::<FROM>(); + let size = std::mem::size_of::<FROM::Native>(); + let offsets = OffsetBuffer::from_repeated_length(size, array.len()); + Ok(Arc::new(GenericBinaryArray::<O>::try_new( + offsets, + array.values().inner().clone(), + array.nulls().cloned(), + )?)) +} + +/// Helper function to cast from one `BinaryArray` or 'LargeBinaryArray' to 'FixedSizeBinaryArray'. +pub(crate) fn cast_binary_to_fixed_size_binary<O: OffsetSizeTrait>( + array: &dyn Array, + byte_width: i32, + cast_options: &CastOptions, +) -> Result<ArrayRef, ArrowError> { + let array = array.as_binary::<O>(); + let mut builder = FixedSizeBinaryBuilder::with_capacity(array.len(), byte_width); + + for i in 0..array.len() { + if array.is_null(i) { + builder.append_null(); + } else { + match builder.append_value(array.value(i)) { + Ok(()) => {} + Err(e) => match cast_options.safe { + true => builder.append_null(), + false => return Err(e), + }, + } + } + } + + Ok(Arc::new(builder.finish())) +} + +/// Helper function to cast from 'FixedSizeBinaryArray' to one `BinaryArray` or 'LargeBinaryArray'. +/// If the target one is too large for the source array it will return an Error. +pub(crate) fn cast_fixed_size_binary_to_binary<O: OffsetSizeTrait>( + array: &dyn Array, + byte_width: i32, +) -> Result<ArrayRef, ArrowError> { + let array = array + .as_any() + .downcast_ref::<FixedSizeBinaryArray>() + .unwrap(); + + let offsets: i128 = byte_width as i128 * array.len() as i128; + + let is_binary = matches!(GenericBinaryType::<O>::DATA_TYPE, DataType::Binary); + if is_binary && offsets > i32::MAX as i128 { + return Err(ArrowError::ComputeError( + "FixedSizeBinary array too large to cast to Binary array".to_string(), + )); + } else if !is_binary && offsets > i64::MAX as i128 { + return Err(ArrowError::ComputeError( + "FixedSizeBinary array too large to cast to LargeBinary array".to_string(), + )); + } + + let mut builder = GenericBinaryBuilder::<O>::with_capacity(array.len(), array.len()); + + for i in 0..array.len() { + if array.is_null(i) { + builder.append_null(); + } else { + builder.append_value(array.value(i)); + } + } + + Ok(Arc::new(builder.finish())) +} + +pub(crate) fn cast_fixed_size_binary_to_binary_view( + array: &dyn Array, + _byte_width: i32, +) -> Result<ArrayRef, ArrowError> { + let array = array + .as_any() + .downcast_ref::<FixedSizeBinaryArray>() + .unwrap(); + + let mut builder = BinaryViewBuilder::with_capacity(array.len()); + for i in 0..array.len() { + if array.is_null(i) { + builder.append_null(); + } else { + builder.append_value(array.value(i)); + } + } + + Ok(Arc::new(builder.finish())) +} + +/// Helper function to cast from one `ByteArrayType` to another and vice versa. +/// If the target one (e.g., `LargeUtf8`) is too large for the source array it will return an Error. +pub(crate) fn cast_byte_container<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError> +where + FROM: ByteArrayType, + TO: ByteArrayType<Native = FROM::Native>, + FROM::Offset: OffsetSizeTrait + ToPrimitive, + TO::Offset: OffsetSizeTrait + NumCast, +{ + let data = array.to_data(); + assert_eq!(data.data_type(), &FROM::DATA_TYPE); + let str_values_buf = data.buffers()[1].clone(); + let offsets = data.buffers()[0].typed_data::<FROM::Offset>(); + + let mut cast_offsets = Vec::<TO::Offset>::with_capacity(offsets.len()); + offsets + .iter() + .try_for_each::<_, Result<_, ArrowError>>(|offset| { + let offset = + <<TO as ByteArrayType>::Offset as NumCast>::from(*offset).ok_or_else(|| { + ArrowError::ComputeError(format!( + "{}{} array too large to cast to {}{} array", + FROM::Offset::PREFIX, + FROM::PREFIX, + TO::Offset::PREFIX, + TO::PREFIX + )) + })?; + cast_offsets.push(offset); + Ok(()) + })?; + + let offset_buffer = Buffer::from_vec(cast_offsets); + + let dtype = TO::DATA_TYPE; + + let builder = ArrayData::builder(dtype) + .offset(array.offset()) + .len(array.len()) + .add_buffer(offset_buffer) + .add_buffer(str_values_buf) + .nulls(data.nulls().cloned()); + + let array_data = unsafe { builder.build_unchecked() }; + + Ok(Arc::new(GenericByteArray::<TO>::from(array_data))) +} + +/// Helper function to cast from one `ByteViewType` array to `ByteArrayType` array. +pub(crate) fn cast_view_to_byte<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError> +where + FROM: ByteViewType, + TO: ByteArrayType, + FROM::Native: AsRef<TO::Native>, +{ + let data = array.to_data(); + let view_array = GenericByteViewArray::<FROM>::from(data); + + let len = view_array.len(); + let bytes = view_array + .views() + .iter() + .map(|v| ByteView::from(*v).length as usize) + .sum::<usize>(); + + let mut byte_array_builder = GenericByteBuilder::<TO>::with_capacity(len, bytes); + + for val in &view_array { + byte_array_builder.append_option(val); + } + + Ok(Arc::new(byte_array_builder.finish())) +} diff --git a/arrow-cast/src/cast/mod.rs b/arrow-cast/src/cast/mod.rs index 4bff879292..0887345d05 100644 --- a/arrow-cast/src/cast/mod.rs +++ b/arrow-cast/src/cast/mod.rs @@ -37,6 +37,7 @@ //! assert_eq!(7.0, c.value(2)); //! ``` +mod binary; mod decimal; mod dictionary; mod list; @@ -46,6 +47,7 @@ mod string; mod structs; mod union; +use crate::cast::binary::*; use crate::cast::decimal::*; use crate::cast::dictionary::*; use crate::cast::list::*; @@ -56,7 +58,6 @@ use crate::cast::structs::*; pub use crate::cast::union::*; use arrow_buffer::IntervalMonthDayNano; -use arrow_data::ByteView; use chrono::{ FixedOffset, LocalResult, NaiveDateTime, NaiveTime, Offset, TimeDelta, TimeZone, Utc, }; @@ -69,7 +70,7 @@ use crate::parse::{ string_to_datetime, }; use arrow_array::{builder::*, cast::*, temporal_conversions::*, timezone::Tz, types::*, *}; -use arrow_buffer::{ArrowNativeType, Buffer, OffsetBuffer, i256}; +use arrow_buffer::{ArrowNativeType, OffsetBuffer, i256}; use arrow_data::ArrayData; use arrow_data::transform::MutableArrayData; use arrow_schema::*; @@ -2632,19 +2633,6 @@ where num_traits::cast::cast::<I, O>(value) } -fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O: OffsetSizeTrait>( - array: &dyn Array, -) -> Result<ArrayRef, ArrowError> { - let array = array.as_primitive::<FROM>(); - let size = std::mem::size_of::<FROM::Native>(); - let offsets = OffsetBuffer::from_repeated_length(size, array.len()); - Ok(Arc::new(GenericBinaryArray::<O>::try_new( - offsets, - array.values().inner().clone(), - array.nulls().cloned(), - )?)) -} - /// Returns the offset to use when interpreting `local` as a wall clock reading /// in `tz`, or `None` if it cannot be resolved. /// @@ -2801,164 +2789,6 @@ where } } -/// Helper function to cast from one `BinaryArray` or 'LargeBinaryArray' to 'FixedSizeBinaryArray'. -fn cast_binary_to_fixed_size_binary<O: OffsetSizeTrait>( - array: &dyn Array, - byte_width: i32, - cast_options: &CastOptions, -) -> Result<ArrayRef, ArrowError> { - let array = array.as_binary::<O>(); - let mut builder = FixedSizeBinaryBuilder::with_capacity(array.len(), byte_width); - - for i in 0..array.len() { - if array.is_null(i) { - builder.append_null(); - } else { - match builder.append_value(array.value(i)) { - Ok(()) => {} - Err(e) => match cast_options.safe { - true => builder.append_null(), - false => return Err(e), - }, - } - } - } - - Ok(Arc::new(builder.finish())) -} - -/// Helper function to cast from 'FixedSizeBinaryArray' to one `BinaryArray` or 'LargeBinaryArray'. -/// If the target one is too large for the source array it will return an Error. -fn cast_fixed_size_binary_to_binary<O: OffsetSizeTrait>( - array: &dyn Array, - byte_width: i32, -) -> Result<ArrayRef, ArrowError> { - let array = array - .as_any() - .downcast_ref::<FixedSizeBinaryArray>() - .unwrap(); - - let offsets: i128 = byte_width as i128 * array.len() as i128; - - let is_binary = matches!(GenericBinaryType::<O>::DATA_TYPE, DataType::Binary); - if is_binary && offsets > i32::MAX as i128 { - return Err(ArrowError::ComputeError( - "FixedSizeBinary array too large to cast to Binary array".to_string(), - )); - } else if !is_binary && offsets > i64::MAX as i128 { - return Err(ArrowError::ComputeError( - "FixedSizeBinary array too large to cast to LargeBinary array".to_string(), - )); - } - - let mut builder = GenericBinaryBuilder::<O>::with_capacity(array.len(), array.len()); - - for i in 0..array.len() { - if array.is_null(i) { - builder.append_null(); - } else { - builder.append_value(array.value(i)); - } - } - - Ok(Arc::new(builder.finish())) -} - -fn cast_fixed_size_binary_to_binary_view( - array: &dyn Array, - _byte_width: i32, -) -> Result<ArrayRef, ArrowError> { - let array = array - .as_any() - .downcast_ref::<FixedSizeBinaryArray>() - .unwrap(); - - let mut builder = BinaryViewBuilder::with_capacity(array.len()); - for i in 0..array.len() { - if array.is_null(i) { - builder.append_null(); - } else { - builder.append_value(array.value(i)); - } - } - - Ok(Arc::new(builder.finish())) -} - -/// Helper function to cast from one `ByteArrayType` to another and vice versa. -/// If the target one (e.g., `LargeUtf8`) is too large for the source array it will return an Error. -fn cast_byte_container<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError> -where - FROM: ByteArrayType, - TO: ByteArrayType<Native = FROM::Native>, - FROM::Offset: OffsetSizeTrait + ToPrimitive, - TO::Offset: OffsetSizeTrait + NumCast, -{ - let data = array.to_data(); - assert_eq!(data.data_type(), &FROM::DATA_TYPE); - let str_values_buf = data.buffers()[1].clone(); - let offsets = data.buffers()[0].typed_data::<FROM::Offset>(); - - let mut cast_offsets = Vec::<TO::Offset>::with_capacity(offsets.len()); - offsets - .iter() - .try_for_each::<_, Result<_, ArrowError>>(|offset| { - let offset = - <<TO as ByteArrayType>::Offset as NumCast>::from(*offset).ok_or_else(|| { - ArrowError::ComputeError(format!( - "{}{} array too large to cast to {}{} array", - FROM::Offset::PREFIX, - FROM::PREFIX, - TO::Offset::PREFIX, - TO::PREFIX - )) - })?; - cast_offsets.push(offset); - Ok(()) - })?; - - let offset_buffer = Buffer::from_vec(cast_offsets); - - let dtype = TO::DATA_TYPE; - - let builder = ArrayData::builder(dtype) - .offset(array.offset()) - .len(array.len()) - .add_buffer(offset_buffer) - .add_buffer(str_values_buf) - .nulls(data.nulls().cloned()); - - let array_data = unsafe { builder.build_unchecked() }; - - Ok(Arc::new(GenericByteArray::<TO>::from(array_data))) -} - -/// Helper function to cast from one `ByteViewType` array to `ByteArrayType` array. -fn cast_view_to_byte<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError> -where - FROM: ByteViewType, - TO: ByteArrayType, - FROM::Native: AsRef<TO::Native>, -{ - let data = array.to_data(); - let view_array = GenericByteViewArray::<FROM>::from(data); - - let len = view_array.len(); - let bytes = view_array - .views() - .iter() - .map(|v| ByteView::from(*v).length as usize) - .sum::<usize>(); - - let mut byte_array_builder = GenericByteBuilder::<TO>::with_capacity(len, bytes); - - for val in &view_array { - byte_array_builder.append_option(val); - } - - Ok(Arc::new(byte_array_builder.finish())) -} - #[cfg(test)] mod tests { use super::*;
