friendlymatthew commented on code in PR #7908:
URL: https://github.com/apache/arrow-rs/pull/7908#discussion_r2202644896
##########
parquet-variant/Cargo.toml:
##########
@@ -37,6 +37,7 @@ arrow-schema = { workspace = true }
chrono = { workspace = true }
indexmap = "2.10.0"
+simdutf8 = { workspace = true }
Review Comment:
Awesome! Any chance we could make this optional?
See https://github.com/apache/arrow-rs/issues/7902#issuecomment-3064697349
##########
parquet-variant/src/utils.rs:
##########
@@ -84,6 +84,21 @@ pub(crate) fn string_from_slice(
.map_err(|_| ArrowError::InvalidArgumentError("invalid UTF-8
string".to_string()))
}
+/// Extracts a byte slice from the given range and validates it as UTF-8.
+#[inline]
+pub(crate) fn extract_and_validate_utf8_slice(
+ bytes: &[u8],
+ range: Range<usize>,
+) -> Result<&str, ArrowError> {
+ let offset_buffer = slice_from_slice(bytes, range)?;
+
+ simdutf8::basic::from_utf8(offset_buffer).map_err(|_| {
+ // Use simdutf8::compat to return details about the decoding error
+ let e = simdutf8::compat::from_utf8(offset_buffer).unwrap_err();
+ ArrowError::InvalidArgumentError(format!("encountered non UTF-8 data:
{e}"))
+ })
+}
+
Review Comment:
I noticed `string_from_slice` is quite similar to this method. I wonder if
we only want one 🤔
If we make `simdutf8` optional, we'd probably need something like
https://github.com/apache/arrow-rs/blob/7b219f98c25fcd318a0c207f51a41398d1b23724/parquet/src/util/utf8.rs#L40-L57
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
