This is an automated email from the ASF dual-hosted git repository.
Jefffrey pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 70219af2ff fix: Consolidate string-to-decimal parsing into a single
code path (#10850)
70219af2ff is described below
commit 70219af2ffa16615e2fcde5760b0218f5c986ac6
Author: Neil Conway <[email protected]>
AuthorDate: Tue Sep 1 11:25:26 2026 -0400
fix: Consolidate string-to-decimal parsing into a single code path (#10850)
# Which issue does this PR close?
- Closes #10787, closes #10788, closes #10789, closes #10790, closes
#10791, closes #10792, closes #10793, closes #10794
# Rationale for this change
arrow-cast had two different string-to-decimal parsers:
`parse_string_to_decimal_native`, used by `cast`, and `parse_decimal`,
used by the CSV and JSON readers. Aside from redundancy, these code
paths behaved differently (e.g., truncating vs rounding for digits
beyond the target type's scale, whitespace trimming, support for
e-notation, etc.), so decimal conversion behaved differently depending
on how the decimal value arrived into arrow-rs.
This PR replaces these parsers with a single unified parser;
`parse_decimal(s, precision_scale)` is now the public entry point and
`parse_string_to_decimal_native` is a thin, deprecated wrapper over it.
The new parser is based on the one-pass, u64-chunked parser in #10668,
extended with support for e-notation and negative scales. This fixes a
lot of bugs and ensures consistent behavior, but it does result in some
behavior changes and a small performance regression for the CSV/JSON
path; more details below.
Bugs fixed: (all in the JSON/CSV path)
- divide-by-zero panic or wrong values for some inputs in exponent
notation (#10788, #5762)
- overflow on inputs with >= 256 digits or long exponents (#10787)
- negative scales were ignored (#10791)
- `0e0`/`-0e0` were incorrectly rejected (#10789), while `e5` and `-.`
were incorrectly accepted (#10790)
Behaviour changes:
- CSV and JSON readers now round half away from zero instead of
truncating digits beyond the scale (#9410, #9422, #7355)
- `cast` from strings accepts exponent notation (#5068) and negative
scales, which `can_cast_types` already advertised (#10792), and
validates the target precision and scale before parsing any values
- CSV and JSON readers now trim whitespace (#10793). Only ASCII
whitespace characters are trimmed, which matches the behavior of the CSV
float/int parsers; previously, the `cast` path trimmed Unicode
whitespace as well, but it will no longer do so.
- parse errors use `ArrowError::ParseError` with unified messages
- `variant_get` validates the target precision for string inputs
(#10794)
Performance:
- `cast` string-to-decimal: ~unchanged. The cast path already used the
fast single-pass parser from #10668; the unified parser benchmarks
within Criterion noise (~4%) of it.
- CSV/JSON reader path: short inputs cost 1-2 ns more per value, while
long Decimal256 inputs are ~35% faster. End-to-end, CSV reads of decimal
columns are ~8% slower. I suspect there is room for further optimization
here (which will now benefit both code paths!) to reach or exceed the
previous performance, but I'd like to land the unified parser first
before we tackle further optimizations.
# What changes are included in this PR?
See above.
# Are these changes tested?
New tests added to cover rounding, exponents, negative scale,
whitespace, long inputs and the four widths, a seeded differential test
checks 20k random inputs against a BigInt reference (num-bigint was
added as a dev-dependency), and the CSV/JSON readers gain end-to-end
tests for the new behavior and bugfixes listed above.
# Are there any user-facing changes?
Yes; a deprecated public API, and user-visible behavioral changes in
decimal parsing.
# AI usage
Iterated primarily with CC Fable 5; code reviewed by Codex GPT 5.6. I
read, understand, and revised the resulting code.
---
Cargo.lock | 1 +
arrow-cast/Cargo.toml | 1 +
arrow-cast/benches/parse_decimal.rs | 72 +-
arrow-cast/src/cast/decimal.rs | 475 ++---------
arrow-cast/src/cast/mod.rs | 35 +-
arrow-cast/src/parse.rs | 1040 ++++++++++++++++++------
arrow-csv/src/reader/mod.rs | 60 +-
arrow-json/src/reader/mod.rs | 94 ++-
parquet-variant-compute/src/type_conversion.rs | 15 +-
9 files changed, 1092 insertions(+), 701 deletions(-)
diff --git a/Cargo.lock b/Cargo.lock
index 8e574cdbb7..9ec5792a23 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -301,6 +301,7 @@ dependencies = [
"half",
"insta",
"lexical-core",
+ "num-bigint 0.5.1",
"num-traits",
"rand 0.10.2",
"ryu",
diff --git a/arrow-cast/Cargo.toml b/arrow-cast/Cargo.toml
index fcaf54b481..ad66ffc3c3 100644
--- a/arrow-cast/Cargo.toml
+++ b/arrow-cast/Cargo.toml
@@ -59,6 +59,7 @@ ryu = "1.0.16"
criterion = { workspace = true, default-features = false }
half = { version = "2.1", default-features = false }
insta = { workspace = true }
+num-bigint = { version = "0.5", default-features = false, features = ["std"] }
rand = "0.10"
[[bench]]
diff --git a/arrow-cast/benches/parse_decimal.rs
b/arrow-cast/benches/parse_decimal.rs
index 93089363bb..bdffcb7cfc 100644
--- a/arrow-cast/benches/parse_decimal.rs
+++ b/arrow-cast/benches/parse_decimal.rs
@@ -15,12 +15,26 @@
// specific language governing permissions and limitations
// under the License.
-use arrow_array::types::{Decimal128Type, Decimal256Type};
-use arrow_cast::cast::parse_string_to_decimal_native;
+use arrow_array::types::{
+ Decimal32Type, Decimal64Type, Decimal128Type, Decimal256Type, DecimalType,
+};
use arrow_cast::parse::parse_decimal;
use criterion::*;
use std::hint;
+fn bench_parse<T: DecimalType>(
+ c: &mut Criterion,
+ name: &str,
+ decimal: &str,
+ precision: u8,
+ scale: i8,
+) {
+ let d = hint::black_box(decimal);
+ c.bench_function(name, |b| {
+ b.iter(|| parse_decimal::<T>(d, precision, scale).unwrap());
+ });
+}
+
fn criterion_benchmark(c: &mut Criterion) {
let decimals = [
"123.123",
@@ -47,13 +61,10 @@ fn criterion_benchmark(c: &mut Criterion) {
];
for decimal in decimals {
- let d = hint::black_box(decimal);
- c.bench_function(d, |b| {
- b.iter(|| parse_decimal::<Decimal256Type>(d, 20, 3).unwrap());
- });
+ bench_parse::<Decimal256Type>(c, decimal, decimal, 20, 3);
}
- let string_decimals = [
+ let decimal128 = [
("string decimal128 integer", "12345678912345678", 3),
("string decimal128 exact scale", "12345678912345.123", 3),
("string decimal128 padded scale", "12345678912345.1", 6),
@@ -64,6 +75,24 @@ fn criterion_benchmark(c: &mut Criterion) {
"99999999999999999999999999999999999999",
0,
),
+ ("string decimal128 exponent", "1.2345678912345e13", 3),
+ (
+ "string decimal128 negative exponent",
+ "12345678912345678e-3",
+ 3,
+ ),
+ ("string decimal128 negative scale", "12345678912345678", -3),
+ (
+ "string decimal128 long fraction",
+ "1.2345678912345678912345678912345678912345",
+ 3,
+ ),
+ ];
+ for (name, decimal, scale) in decimal128 {
+ bench_parse::<Decimal128Type>(c, name, decimal, 38, scale);
+ }
+
+ let decimal256 = [
(
"string decimal256 76 digits",
"9999999999999999999999999999999999999999999999999999999999999999999999999999",
@@ -75,19 +104,24 @@ fn criterion_benchmark(c: &mut Criterion) {
3,
),
];
+ for (name, decimal, scale) in decimal256 {
+ bench_parse::<Decimal256Type>(c, name, decimal, 76, scale);
+ }
+
+ let decimal32 = [
+ ("string decimal32 short", "1234.56", 2),
+ ("string decimal32 9 digits", "9999999.99", 2),
+ ];
+ for (name, decimal, scale) in decimal32 {
+ bench_parse::<Decimal32Type>(c, name, decimal, 9, scale);
+ }
- for (name, decimal, scale) in string_decimals {
- let d = hint::black_box(decimal);
- let scale = hint::black_box(scale);
- if name.contains("decimal256") {
- c.bench_function(name, |b| {
- b.iter(|| parse_string_to_decimal_native::<Decimal256Type>(d,
scale).unwrap());
- });
- } else {
- c.bench_function(name, |b| {
- b.iter(|| parse_string_to_decimal_native::<Decimal128Type>(d,
scale).unwrap());
- });
- }
+ let decimal64 = [
+ ("string decimal64 short", "1234.56", 2),
+ ("string decimal64 18 digits", "9999999999999999.99", 2),
+ ];
+ for (name, decimal, scale) in decimal64 {
+ bench_parse::<Decimal64Type>(c, name, decimal, 18, scale);
}
}
diff --git a/arrow-cast/src/cast/decimal.rs b/arrow-cast/src/cast/decimal.rs
index 574133746e..a5d704bba8 100644
--- a/arrow-cast/src/cast/decimal.rs
+++ b/arrow-cast/src/cast/decimal.rs
@@ -16,6 +16,7 @@
// under the License.
use crate::cast::*;
+use crate::parse::{DecimalParseError, parse_decimal_checked};
/// A utility trait that provides checked conversions between
/// decimal types inspired by [`NumCast`]
@@ -532,176 +533,31 @@ where
/// unscaled representation in the decimal type's native integer (e.g. `i32`
/// for `Decimal32Type`, `i256` for `Decimal256Type`).
///
-/// The input is an optionally signed (`+`/`-`) sequence of digits containing
-/// at most one decimal point, with optional surrounding whitespace. Fractional
-/// digits beyond `scale` do not appear in the result but round it half away
-/// from zero (e.g. `"1.005"` at scale 2 parses as `101`).
-///
/// Returns an error if the input is not a valid decimal string, or if the
-/// scaled and rounded value overflows the native type. The caller is
-/// responsible for validating the result against a precision.
+/// scaled and rounded value does not fit the maximum precision of the decimal
+/// type. The caller is responsible for validating the result against any
+/// smaller target precision.
+#[deprecated(
+ since = "60.0.0",
+ note = "Use `arrow_cast::parse::parse_decimal` instead"
+)]
pub fn parse_string_to_decimal_native<T: DecimalType>(
value_str: &str,
scale: usize,
-) -> Result<T::Native, ArrowError>
-where
- T::Native: DecimalCast + ArrowNativeTypeOp,
-{
- let value_str = value_str.trim();
- let bytes = value_str.as_bytes();
-
- let mut index = 0;
- let negative = match bytes.first() {
- Some(b'-') => {
- index += 1;
- true
- }
- Some(b'+') => {
- index += 1;
- false
- }
- _ => false,
+) -> Result<T::Native, ArrowError> {
+ let overflow = || {
+ ArrowError::InvalidArgumentError(format!(
+ "Cannot convert {value_str} to {}: Overflow",
+ T::PREFIX
+ ))
};
-
- let mut value = T::Native::ZERO;
- let mut chunk = 0_u64;
- let mut chunk_len = 0_usize;
- let mut saw_digit = false;
- let mut saw_point = false;
- let mut fractionals = 0_usize;
- let mut first_discarded_digit = None;
-
- while let Some(&b) = bytes.get(index) {
- match b {
- b'0'..=b'9' => {
- saw_digit = true;
- let digit = b - b'0';
- if saw_point {
- if fractionals == scale {
- first_discarded_digit.get_or_insert(digit);
- index += 1;
- continue;
- }
- fractionals += 1;
- }
-
- // Cannot overflow: the chunk is folded into `value` before it
- // exceeds MAX_CHUNK_DIGITS digits, all of which fit in a u64
- chunk = chunk * 10 + digit as u64;
- chunk_len += 1;
- if chunk_len == MAX_CHUNK_DIGITS {
- value = fold_decimal_chunk::<T>(value, chunk, chunk_len,
negative, value_str)?;
- chunk = 0;
- chunk_len = 0;
- }
- }
- b'.' if !saw_point => saw_point = true,
- _ => {
- return Err(ArrowError::InvalidArgumentError(format!(
- "Invalid decimal format: {value_str:?}"
- )));
- }
- }
- index += 1;
- }
-
- if chunk_len > 0 {
- value = fold_decimal_chunk::<T>(value, chunk, chunk_len, negative,
value_str)?;
- }
-
- if !saw_digit {
- return Err(ArrowError::InvalidArgumentError(format!(
- "Invalid decimal format: {value_str:?}"
- )));
- }
-
- // Scale the value up to the target scale. Skipped for zero, where
computing
- // 10^(scale - fractionals) could overflow the native type even though the
- // result (zero) is always representable.
- if fractionals < scale && !value.is_zero() {
- value = value
- .mul_checked(decimal_pow::<T>(scale - fractionals, value_str)?)
- .map_err(|_| decimal_parse_overflow::<T>(value_str))?;
- }
-
- if first_discarded_digit.is_some_and(|digit| digit >= 5) {
- value = if negative {
- value.sub_checked(T::Native::ONE)
- } else {
- value.add_checked(T::Native::ONE)
+ let scale = i8::try_from(scale).map_err(|_| overflow())?;
+ parse_decimal_checked::<T>(value_str, T::MAX_PRECISION, scale).map_err(|e|
match e {
+ DecimalParseError::InvalidFormat => {
+ ArrowError::InvalidArgumentError(format!("Invalid decimal format:
{value_str:?}"))
}
- .map_err(|_| decimal_parse_overflow::<T>(value_str))?;
- }
-
- Ok(value)
-}
-
-/// The maximum number of decimal digits a u64 can accumulate without
-/// overflowing: every 19-digit number fits in a u64.
-const MAX_CHUNK_DIGITS: usize = 19;
-
-/// Folds a chunk of up to [`MAX_CHUNK_DIGITS`] digits into `value`, producing
-/// `value * 10^chunk_len + chunk` (`chunk` is negated first when parsing a
-/// negative number).
-#[inline]
-fn fold_decimal_chunk<T: DecimalType>(
- value: T::Native,
- chunk: u64,
- chunk_len: usize,
- negative: bool,
- value_str: &str,
-) -> Result<T::Native, ArrowError>
-where
- T::Native: DecimalCast + ArrowNativeTypeOp,
-{
- // Negate before narrowing to the native type so that a chunk with the
- // magnitude of the native type's minimum value (e.g. "2147483648" for
- // Decimal32) remains representable.
- let signed_chunk = if negative {
- -(chunk as i128)
- } else {
- chunk as i128
- };
- let chunk = T::Native::from_decimal(signed_chunk)
- .ok_or_else(|| decimal_parse_overflow::<T>(value_str))?;
-
- // When `value` is zero the multiply would be a no-op; skipping it avoids
- // computing 10^chunk_len, which can overflow a narrow native type even
- // though the result (the chunk itself) is representable.
- if value.is_zero() {
- return Ok(chunk);
- }
-
- value
- .mul_checked(decimal_pow::<T>(chunk_len, value_str)?)
- .map_err(|_| decimal_parse_overflow::<T>(value_str))?
- .add_checked(chunk)
- .map_err(|_| decimal_parse_overflow::<T>(value_str))
-}
-
-/// Returns `10^exp` as a `T::Native`, or an overflow error if the result does
-/// not fit in the native type.
-#[inline]
-fn decimal_pow<T: DecimalType>(exp: usize, value_str: &str) ->
Result<T::Native, ArrowError>
-where
- T::Native: ArrowNativeTypeOp,
-{
- // T::MAX_FOR_EACH_PRECISION[k] holds 10^k - 1, so adding one yields 10^k
- // without computing a power at runtime. Exponents beyond the table always
- // overflow: the native type cannot hold 10^(MAX_PRECISION + 1).
- let max = T::MAX_FOR_EACH_PRECISION
- .get(exp)
- .ok_or_else(|| decimal_parse_overflow::<T>(value_str))?;
- Ok(max.add_wrapping(T::Native::ONE))
-}
-
-#[inline]
-fn decimal_parse_overflow<T: DecimalType>(value_str: &str) -> ArrowError {
- ArrowError::InvalidArgumentError(format!(
- "Cannot convert {} to {}: Overflow",
- value_str,
- T::PREFIX
- ))
+ DecimalParseError::Overflow => overflow(),
+ })
}
pub(crate) fn generic_string_to_decimal_cast<'a, T, S>(
@@ -712,14 +568,12 @@ pub(crate) fn generic_string_to_decimal_cast<'a, T, S>(
) -> Result<PrimitiveArray<T>, ArrowError>
where
T: DecimalType,
- T::Native: DecimalCast + ArrowNativeTypeOp,
&'a S: StringArrayType<'a>,
{
if cast_options.safe {
- let iter = from.iter().map(|v| {
- let v = v.and_then(|v| parse_string_to_decimal_native::<T>(v,
scale as usize).ok())?;
- T::is_valid_decimal_precision(v, precision).then_some(v)
- });
+ let iter = from
+ .iter()
+ .map(|v| parse_decimal_checked::<T>(v?, precision, scale).ok());
// Benefit:
// 15-19% faster than appending to a PrimitiveBuilder (measured
// with the cast_kernels string-to-decimal benchmarks)
@@ -734,16 +588,16 @@ where
for v in from.iter() {
match v {
Some(v) => {
- let v = parse_string_to_decimal_native::<T>(v, scale as
usize)
- .map_err(|e| {
- ArrowError::CastError(format!(
- "Cannot cast string '{v}' to value of
{}({precision}, {scale}) type: {e}",
- T::PREFIX,
- ))
- })
- .and_then(|v| {
- T::validate_decimal_precision(v, precision,
scale).map(|()| v)
- })?;
+ let v = parse_decimal_checked::<T>(v, precision,
scale).map_err(|e| {
+ let reason = match e {
+ DecimalParseError::InvalidFormat => "invalid
decimal format",
+ DecimalParseError::Overflow => "value does not
fit",
+ };
+ ArrowError::CastError(format!(
+ "Cannot cast string '{v}' to value of
{}({precision}, {scale}) type: {reason}",
+ T::PREFIX,
+ ))
+ })?;
builder.append_value(v);
}
None => builder.append_null(),
@@ -753,16 +607,12 @@ where
}
}
-pub(crate) fn string_to_decimal_cast<T, Offset: OffsetSizeTrait>(
+pub(crate) fn string_to_decimal_cast<T: DecimalType, Offset: OffsetSizeTrait>(
from: &GenericStringArray<Offset>,
precision: u8,
scale: i8,
cast_options: &CastOptions,
-) -> Result<PrimitiveArray<T>, ArrowError>
-where
- T: DecimalType,
- T::Native: DecimalCast + ArrowNativeTypeOp,
-{
+) -> Result<PrimitiveArray<T>, ArrowError> {
generic_string_to_decimal_cast::<T, GenericStringArray<Offset>>(
from,
precision,
@@ -771,42 +621,23 @@ where
)
}
-pub(crate) fn string_view_to_decimal_cast<T>(
+pub(crate) fn string_view_to_decimal_cast<T: DecimalType>(
from: &StringViewArray,
precision: u8,
scale: i8,
cast_options: &CastOptions,
-) -> Result<PrimitiveArray<T>, ArrowError>
-where
- T: DecimalType,
- T::Native: DecimalCast + ArrowNativeTypeOp,
-{
+) -> Result<PrimitiveArray<T>, ArrowError> {
generic_string_to_decimal_cast::<T, StringViewArray>(from, precision,
scale, cast_options)
}
/// Cast Utf8 to decimal
-pub(crate) fn cast_string_to_decimal<T, Offset: OffsetSizeTrait>(
+pub(crate) fn cast_string_to_decimal<T: DecimalType, Offset: OffsetSizeTrait>(
from: &dyn Array,
precision: u8,
scale: i8,
cast_options: &CastOptions,
-) -> Result<ArrayRef, ArrowError>
-where
- T: DecimalType,
- T::Native: DecimalCast + ArrowNativeTypeOp,
-{
- if scale < 0 {
- return Err(ArrowError::InvalidArgumentError(format!(
- "Cannot cast string to decimal with negative scale {scale}"
- )));
- }
-
- if scale > T::MAX_SCALE {
- return Err(ArrowError::InvalidArgumentError(format!(
- "Cannot cast string to decimal greater than maximum scale {}",
- T::MAX_SCALE
- )));
- }
+) -> Result<ArrayRef, ArrowError> {
+ validate_decimal_precision_and_scale::<T>(precision, scale)?;
let result = match from.data_type() {
DataType::Utf8View => string_view_to_decimal_cast::<T>(
@@ -1008,225 +839,33 @@ mod tests {
use super::*;
#[test]
- fn test_parse_string_to_decimal_native() -> Result<(), ArrowError> {
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("0", 0)?,
- 0_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("0", 5)?,
- 0_i128
- );
-
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("123", 0)?,
- 123_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("123", 5)?,
- 12300000_i128
- );
-
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("123.45", 0)?,
- 123_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("123.45", 5)?,
- 12345000_i128
- );
-
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("123.4567891",
0)?,
- 123_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("123.4567891",
5)?,
- 12345679_i128
- );
- Ok(())
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_integer_widths() -> Result<(),
ArrowError> {
- assert_eq!(
- parse_string_to_decimal_native::<Decimal32Type>("123.45", 2)?,
- 12_345_i32
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal32Type>("-2147483648", 0)?,
- i32::MIN
- );
- assert!(parse_string_to_decimal_native::<Decimal32Type>("2147483648",
0).is_err());
-
- assert_eq!(
- parse_string_to_decimal_native::<Decimal64Type>("123.45", 2)?,
- 12_345_i64
- );
- assert_eq!(
-
parse_string_to_decimal_native::<Decimal64Type>("-9223372036854775808", 0)?,
- i64::MIN
- );
-
assert!(parse_string_to_decimal_native::<Decimal64Type>("9223372036854775808",
0).is_err());
-
- assert_eq!(
-
parse_string_to_decimal_native::<Decimal128Type>(&i128::MAX.to_string(), 0)?,
- i128::MAX
- );
- assert_eq!(
-
parse_string_to_decimal_native::<Decimal128Type>(&i128::MIN.to_string(), 0)?,
- i128::MIN
- );
-
- assert_eq!(
-
parse_string_to_decimal_native::<Decimal256Type>(&i256::MAX.to_string(), 0)?,
- i256::MAX
- );
- assert_eq!(
-
parse_string_to_decimal_native::<Decimal256Type>(&i256::MIN.to_string(), 0)?,
- i256::MIN
- );
- Ok(())
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_rounding_and_padding() ->
Result<(), ArrowError> {
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("12.", 2)?,
- 1_200_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>(".12", 2)?,
- 12_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("+.12", 2)?,
- 12_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("-.12", 2)?,
- -12_i128
- );
+ #[expect(deprecated)]
+ fn test_parse_string_to_decimal_native() {
assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>(".5", 0)?,
- 1_i128
+ parse_string_to_decimal_native::<Decimal128Type>("123.456",
2).unwrap(),
+ 12346
);
+ // The value is checked against the maximum precision of the type, not
+ // against any target precision
assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("-.5", 0)?,
- -1_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("1.234", 2)?,
- 123_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("1.235", 2)?,
- 124_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("-1.234", 2)?,
- -123_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("-1.235", 2)?,
- -124_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("-0.004", 2)?,
- 0_i128
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("-0.005", 2)?,
- -1_i128
- );
- Ok(())
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_rounding_overflow() {
-
assert!(parse_string_to_decimal_native::<Decimal32Type>("2147483647.5",
0).is_err());
-
assert!(parse_string_to_decimal_native::<Decimal32Type>("-2147483648.5",
0).is_err());
-
- assert!(
- parse_string_to_decimal_native::<Decimal128Type>(&format!("{}.5",
i128::MAX), 0)
- .is_err()
- );
- assert!(
- parse_string_to_decimal_native::<Decimal128Type>(&format!("{}.5",
i128::MIN), 0)
- .is_err()
- );
-
- assert!(
- parse_string_to_decimal_native::<Decimal256Type>(&format!("{}.5",
i256::MAX), 0)
- .is_err()
+ parse_string_to_decimal_native::<Decimal128Type>(&"9".repeat(38),
0).unwrap(),
+ 10_i128.pow(38) - 1
);
assert!(
- parse_string_to_decimal_native::<Decimal256Type>(&format!("{}.5",
i256::MIN), 0)
- .is_err()
- );
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_overflow_not_wrapped() {
- // The unscaled value (integer digits scaled by 10^21) far exceeds the
- // i256 range, so this must report overflow rather than wrapping to an
- // arbitrary (possibly in-range) value
- let input = format!("{}.12345678901234567890123", "7".repeat(71));
- assert!(parse_string_to_decimal_native::<Decimal256Type>(&input,
21).is_err());
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_long_fraction() -> Result<(),
ArrowError> {
- // Fractional parts longer than any native integer type parse fine;
- // digits beyond the scale only matter for rounding
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>(&format!(".{}",
"1".repeat(100)), 4)?,
- 1_111_i128
+
parse_string_to_decimal_native::<Decimal128Type>(&i128::MAX.to_string(),
0).is_err()
);
assert_eq!(
- parse_string_to_decimal_native::<Decimal64Type>(&format!(".{}",
"5".repeat(100)), 4)?,
- 5_556_i64
+ parse_string_to_decimal_native::<Decimal128Type>("abc", 2)
+ .unwrap_err()
+ .to_string(),
+ "Invalid argument error: Invalid decimal format: \"abc\""
);
- Ok(())
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_zero_with_large_scale() ->
Result<(), ArrowError> {
- // 10^scale overflows the native type, but zero is still representable
assert_eq!(
- parse_string_to_decimal_native::<Decimal32Type>("0", 10)?,
- 0_i32
+ parse_string_to_decimal_native::<Decimal32Type>("1", 10)
+ .unwrap_err()
+ .to_string(),
+ "Invalid argument error: Cannot convert 1 to Decimal32: Overflow"
);
- assert_eq!(
- parse_string_to_decimal_native::<Decimal32Type>("-0.0", 10)?,
- 0_i32
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal64Type>("0", 20)?,
- 0_i64
- );
- assert_eq!(
- parse_string_to_decimal_native::<Decimal128Type>("0", 40)?,
- 0_i128
- );
- assert!(parse_string_to_decimal_native::<Decimal32Type>("1",
10).is_err());
- Ok(())
- }
-
- #[test]
- fn test_parse_string_to_decimal_native_invalid_syntax() {
- for input in [
- "", " ", ".", "+", "-", "+.", "-.", "1.2.3", "1e2", "1.-2", "--1",
- ] {
- assert!(
- parse_string_to_decimal_native::<Decimal128Type>(input,
2).is_err(),
- "expected {input:?} to fail parsing as Decimal128"
- );
- assert!(
- parse_string_to_decimal_native::<Decimal256Type>(input,
2).is_err(),
- "expected {input:?} to fail parsing as Decimal256"
- );
- }
}
#[test]
diff --git a/arrow-cast/src/cast/mod.rs b/arrow-cast/src/cast/mod.rs
index 459b47ac1c..a63d7585ed 100644
--- a/arrow-cast/src/cast/mod.rs
+++ b/arrow-cast/src/cast/mod.rs
@@ -72,9 +72,9 @@ use arrow_schema::*;
use arrow_select::take::take;
use num_traits::{NumCast, ToPrimitive, cast::AsPrimitive};
-pub use decimal::{
- DecimalCast, parse_string_to_decimal_native, rescale_decimal,
single_float_to_decimal,
-};
+#[expect(deprecated)]
+pub use decimal::parse_string_to_decimal_native;
+pub use decimal::{DecimalCast, rescale_decimal, single_float_to_decimal};
pub use string::cast_single_string_to_boolean_default;
/// Lossy conversion from decimal to float.
@@ -2895,6 +2895,7 @@ where
#[cfg(test)]
mod tests {
use super::*;
+ use crate::parse::parse_decimal;
use DataType::*;
use arrow_array::{Int64Array, RunArray, StringArray};
use arrow_buffer::{Buffer, IntervalDayTime, NullBuffer};
@@ -11139,10 +11140,10 @@ mod tests {
}
#[test]
- fn test_parse_string_to_decimal() {
+ fn test_parse_decimal_and_format() {
assert_eq!(
Decimal128Type::format_decimal(
- parse_string_to_decimal_native::<Decimal128Type>("123.45",
2).unwrap(),
+ parse_decimal::<Decimal128Type>("123.45", 38, 2).unwrap(),
38,
2,
),
@@ -11150,7 +11151,7 @@ mod tests {
);
assert_eq!(
Decimal128Type::format_decimal(
- parse_string_to_decimal_native::<Decimal128Type>("12345",
2).unwrap(),
+ parse_decimal::<Decimal128Type>("12345", 38, 2).unwrap(),
38,
2,
),
@@ -11158,7 +11159,7 @@ mod tests {
);
assert_eq!(
Decimal128Type::format_decimal(
- parse_string_to_decimal_native::<Decimal128Type>("0.12345",
2).unwrap(),
+ parse_decimal::<Decimal128Type>("0.12345", 38, 2).unwrap(),
38,
2,
),
@@ -11166,7 +11167,7 @@ mod tests {
);
assert_eq!(
Decimal128Type::format_decimal(
- parse_string_to_decimal_native::<Decimal128Type>(".12345",
2).unwrap(),
+ parse_decimal::<Decimal128Type>(".12345", 38, 2).unwrap(),
38,
2,
),
@@ -11174,7 +11175,7 @@ mod tests {
);
assert_eq!(
Decimal128Type::format_decimal(
- parse_string_to_decimal_native::<Decimal128Type>(".1265",
2).unwrap(),
+ parse_decimal::<Decimal128Type>(".1265", 38, 2).unwrap(),
38,
2,
),
@@ -11182,7 +11183,7 @@ mod tests {
);
assert_eq!(
Decimal128Type::format_decimal(
- parse_string_to_decimal_native::<Decimal128Type>(".1265",
2).unwrap(),
+ parse_decimal::<Decimal128Type>(".1265", 38, 2).unwrap(),
38,
2,
),
@@ -11191,7 +11192,7 @@ mod tests {
assert_eq!(
Decimal256Type::format_decimal(
- parse_string_to_decimal_native::<Decimal256Type>("123.45",
3).unwrap(),
+ parse_decimal::<Decimal256Type>("123.45", 76, 3).unwrap(),
38,
3,
),
@@ -11199,7 +11200,7 @@ mod tests {
);
assert_eq!(
Decimal256Type::format_decimal(
- parse_string_to_decimal_native::<Decimal256Type>("12345",
3).unwrap(),
+ parse_decimal::<Decimal256Type>("12345", 76, 3).unwrap(),
38,
3,
),
@@ -11207,7 +11208,7 @@ mod tests {
);
assert_eq!(
Decimal256Type::format_decimal(
- parse_string_to_decimal_native::<Decimal256Type>("0.12345",
3).unwrap(),
+ parse_decimal::<Decimal256Type>("0.12345", 76, 3).unwrap(),
38,
3,
),
@@ -11215,7 +11216,7 @@ mod tests {
);
assert_eq!(
Decimal256Type::format_decimal(
- parse_string_to_decimal_native::<Decimal256Type>(".12345",
3).unwrap(),
+ parse_decimal::<Decimal256Type>(".12345", 76, 3).unwrap(),
38,
3,
),
@@ -11223,7 +11224,7 @@ mod tests {
);
assert_eq!(
Decimal256Type::format_decimal(
- parse_string_to_decimal_native::<Decimal256Type>(".1265",
3).unwrap(),
+ parse_decimal::<Decimal256Type>(".1265", 76, 3).unwrap(),
38,
3,
),
@@ -11556,7 +11557,7 @@ mod tests {
},
);
assert_eq!(
- "Invalid argument error: 1000.00000000 is too large to store in a
Decimal128 of precision 10. Max is 99.99999999",
+ "Cast error: Cannot cast string '1000' to value of Decimal128(10,
8) type: value does not fit",
err.unwrap_err().to_string()
);
}
@@ -11642,7 +11643,7 @@ mod tests {
},
);
assert_eq!(
- "Invalid argument error: 1000.00000000 is too large to store in a
Decimal256 of precision 10. Max is 99.99999999",
+ "Cast error: Cannot cast string '1000' to value of Decimal256(10,
8) type: value does not fit",
err.unwrap_err().to_string()
);
}
diff --git a/arrow-cast/src/parse.rs b/arrow-cast/src/parse.rs
index 1b691a8cc8..f2bcbebf19 100644
--- a/arrow-cast/src/parse.rs
+++ b/arrow-cast/src/parse.rs
@@ -796,249 +796,352 @@ impl Parser for Date64Type {
}
}
-fn parse_e_notation<T: DecimalType>(
+/// Parses the string representation of a decimal number into the unscaled
+/// native value of a decimal type with the given `precision` and `scale`.
+///
+/// The accepted syntax is:
+///
+/// ```text
+/// [whitespace] [+|-] digits [. [digits]] [(e|E) [+|-] digits] [whitespace]
+/// ```
+///
+/// or the same with the integer digits omitted (e.g. `.5`), as long as at
+/// least one digit is present in the mantissa. ASCII whitespace is trimmed
+/// from both ends. The exponent is applied before scaling, so `1.5e2` and
+/// `150` parse identically.
+///
+/// Fractional digits beyond `scale` are not stored but round the result half
+/// away from zero (e.g. `1.005` at scale 2 is `101`, `-1.005` is `-101`).
+/// Negative scales are supported and round the integer part in the same way
+/// (e.g. `150` at scale -2 is `2`).
+///
+/// Returns an error if the input is not a valid decimal string, or if the
+/// result does not fit the given precision.
+///
+/// # Example
+///
+/// ```
+/// # use arrow_array::types::Decimal128Type;
+/// # use arrow_cast::parse::parse_decimal;
+/// assert_eq!(parse_decimal::<Decimal128Type>("123.45", 10, 2).unwrap(),
12345);
+/// assert_eq!(parse_decimal::<Decimal128Type>("1.005", 10, 2).unwrap(), 101);
+/// assert_eq!(parse_decimal::<Decimal128Type>("1.5e2", 10, 0).unwrap(), 150);
+/// assert!(parse_decimal::<Decimal128Type>("1234.5", 5, 2).is_err()); // does
not fit
+/// ```
+pub fn parse_decimal<T: DecimalType>(
s: &str,
- mut digits: u16,
- mut fractionals: i16,
- mut result: T::Native,
- index: usize,
- precision: u16,
- scale: i16,
+ precision: u8,
+ scale: i8,
) -> Result<T::Native, ArrowError> {
- let mut exp: i16 = 0;
- let base = T::Native::usize_as(10);
-
- // e has a plus sign
- let mut pos_shift_direction = true;
-
- // skip to the exponent index directly or just after any processed
fractionals
- let mut bs = s.as_bytes().iter().skip(index + fractionals as usize);
-
- // This function is only called from `parse_decimal`, in which we skip
parsing any fractionals
- // after we reach `scale` digits, not knowing ahead of time whether the
decimal contains an
- // e-notation or not.
- // So once we do hit into an e-notation, and drop down into this function,
we need to parse the
- // remaining unprocessed fractionals too, since otherwise we might lose
precision.
- for b in bs.by_ref() {
- match b {
- b'0'..=b'9' => {
- result = result.mul_wrapping(base);
- result = result.add_wrapping(T::Native::usize_as((b - b'0') as
usize));
- fractionals += 1;
- digits += 1;
- }
- b'e' | b'E' => {
- break;
- }
- _ => {
- return Err(ArrowError::ParseError(format!(
- "can't parse the string value {s} to decimal"
- )));
- }
- }
- }
-
- // parse the exponent itself
- let mut signed = false;
- for b in bs {
- match b {
- b'-' if !signed => {
- pos_shift_direction = false;
- signed = true;
- }
- b'+' if !signed => {
- pos_shift_direction = true;
- signed = true;
- }
- b if b.is_ascii_digit() => {
- exp *= 10;
- exp += (b - b'0') as i16;
- }
- _ => {
- return Err(ArrowError::ParseError(format!(
- "can't parse the string value {s} to decimal"
- )));
- }
+ parse_decimal_checked::<T>(s, precision, scale).map_err(|e| match e {
+ DecimalParseError::Overflow => ArrowError::ParseError(format!(
+ "{s:?} does not fit in {}({precision}, {scale})",
+ T::PREFIX
+ )),
+ DecimalParseError::InvalidFormat => {
+ ArrowError::ParseError(format!("Invalid decimal format: {s:?}"))
}
- }
+ })
+}
- if digits == 0 && fractionals == 0 && exp == 0 {
- return Err(ArrowError::ParseError(format!(
- "can't parse the string value {s} to decimal"
- )));
- }
+/// The reason a decimal string could not be parsed.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum DecimalParseError {
+ /// The input is not a valid decimal string
+ InvalidFormat,
+ /// The value does not fit in the precision or the native type of the
decimal
+ Overflow,
+}
- if !pos_shift_direction {
- // exponent has a large negative sign
- // 1.12345e-30 => 0.0{29}12345, scale = 5
- if exp - (digits as i16 + scale) > 0 {
- return Ok(T::Native::usize_as(0));
- }
- exp *= -1;
+/// Like [`parse_decimal`], but reports failures as a [`DecimalParseError`]
+/// instead of formatting an error message, for callers that discard or
+/// re-wrap the error.
+pub(crate) fn parse_decimal_checked<T: DecimalType>(
+ s: &str,
+ precision: u8,
+ scale: i8,
+) -> Result<T::Native, DecimalParseError> {
+ let value = parse_decimal_native::<T>(s, scale)?;
+ if T::is_valid_decimal_precision(value, precision) {
+ Ok(value)
+ } else {
+ Err(DecimalParseError::Overflow)
}
+}
- // point offset
- exp = fractionals - exp;
- // We have zeros on the left, we need to count them
- if !pos_shift_direction && exp > digits as i16 {
- digits = exp as u16;
- }
- // Number of numbers to be removed or added
- exp = scale - exp;
+/// Parses `s` as a decimal with the given `scale` into the native type of `T`,
+/// checking only that the result fits the native type (not the precision).
+///
+/// See [`parse_decimal`] for the accepted syntax and rounding behaviour.
+fn parse_decimal_native<T: DecimalType>(
+ s: &str,
+ scale: i8,
+) -> Result<T::Native, DecimalParseError> {
+ let bytes = s.as_bytes().trim_ascii();
+ let (negative, mut mantissa) = split_sign(bytes);
+
+ let mut scale = scale as i64;
+ loop {
+ let exponent_at = match parse_decimal_mantissa::<T>(mantissa,
negative, scale) {
+ Ok(value) => return Ok(value),
+ Err(MantissaError::InvalidFormat) => return
Err(DecimalParseError::InvalidFormat),
+ Err(MantissaError::Exponent(index)) => index,
+ // The digits before an exponent marker need not fit on their own
+ // (e.g. "4825037936439135476E-14"), so the overflow only stands
+ // if no marker follows
+ Err(MantissaError::Overflow) => mantissa
+ .iter()
+ .position(|b| matches!(b, b'e' | b'E'))
+ .ok_or(DecimalParseError::Overflow)?,
+ };
- if (digits as i16 + exp) as u16 > precision {
- return Err(ArrowError::ParseError(format!(
- "parse decimal overflow ({s})"
- )));
- }
+ // If we saw an exponent, update the effective scale and rescan.
+ // Exponents are rare, so the risk of repeated work is preferable to
+ // the cost of scanning ahead for an exponent marker for every input.
+ let exponent = parse_decimal_exponent(&mantissa[exponent_at + 1..])?;
+ scale = scale.saturating_add(exponent);
- if exp < 0 {
- result = result.div_wrapping(base.pow_wrapping(-exp as _));
- } else {
- result = result.mul_wrapping(base.pow_wrapping(exp as _));
+ // Trim the exponent so the next iteration succeeds without rescanning
+ mantissa = &mantissa[..exponent_at];
}
+}
- Ok(result)
+/// Why scanning the digits of a decimal string stopped.
+enum MantissaError {
+ /// The input is not a valid decimal string
+ InvalidFormat,
+ /// The value does not fit in the native type
+ Overflow,
+ /// An exponent marker (`e` or `E`) was found at the given byte offset
+ Exponent(usize),
}
-/// Parse the string format decimal value to i128/i256 format and checking the
precision and scale.
-/// Expected behavior:
-/// - The result value can't be out of bounds.
-/// - When parsing a decimal with scale 0, all fractional digits will be
discarded. The final
-/// fractional digits may be a subset or a superset of the digits after the
decimal point when
-/// e-notation is used.
-pub fn parse_decimal<T: DecimalType>(
- s: &str,
- precision: u8,
- scale: i8,
-) -> Result<T::Native, ArrowError> {
- let mut result = T::Native::usize_as(0);
- let mut fractionals: i8 = 0;
- let mut digits: u8 = 0;
- let base = T::Native::usize_as(10);
-
- let bs = s.as_bytes();
-
- if !bs
- .last()
- .is_some_and(|b| b.is_ascii_digit() || (b == &b'.' && s.len() > 1))
- {
- // If the last character is not a digit (or a decimal point prefixed
with some digits), then
- // it's not a valid decimal.
- return Err(ArrowError::ParseError(format!(
- "can't parse the string value {s} to decimal"
- )));
+impl From<DecimalParseError> for MantissaError {
+ fn from(e: DecimalParseError) -> Self {
+ match e {
+ DecimalParseError::InvalidFormat => Self::InvalidFormat,
+ DecimalParseError::Overflow => Self::Overflow,
+ }
}
+}
- let (signed, negative) = match bs.first() {
- Some(b'-') => (true, true),
- Some(b'+') => (true, false),
- _ => (false, false),
- };
+/// The maximum number of decimal digits accumulated in a `u64` before the
+/// chunk is folded into the native value: every 18-digit number fits in a
+/// `u64`, and splits into two halves that each fit in a `u32` (see
+/// [`decimal_chunk_to_native`]).
+const MAX_CHUNK_DIGITS: usize = 18;
- // Iterate over the raw input bytes, skipping the sign if any
- let mut bs = bs.iter().enumerate().skip(signed as usize);
+/// Scans `mantissa` (digits with at most one decimal point; the sign has
+/// already been removed) and folds the digits that are significant at
+/// `scale` into a native value, rounding half away from zero on the first
+/// digit that is not.
+#[inline]
+fn parse_decimal_mantissa<T: DecimalType>(
+ mantissa: &[u8],
+ negative: bool,
+ scale: i64,
+) -> Result<T::Native, MantissaError> {
+ // The number of integer and fractional digits that contribute to the
+ // result. For a non-negative scale that is every integer digit and the
+ // first `scale` fractional digits. For a negative scale the last `-scale`
+ // integer digits (and every fractional digit) only matter for rounding.
+ let (int_keep, frac_keep, mut round) = if scale >= 0 {
+ (
+ usize::MAX,
+ usize::try_from(scale).unwrap_or(usize::MAX),
+ true,
+ )
+ } else {
+ let int_digits = mantissa.iter().take_while(|b|
b.is_ascii_digit()).count();
+ match usize::try_from(int_digits as i64 + scale) {
+ Ok(keep) => (keep, 0, true),
+ // Even the first digit is more than one position below the
+ // least significant digit of the result: the value rounds to
+ // zero regardless of what the digits are
+ Err(_) => (0, 0, false),
+ }
+ };
- let mut is_e_notation = false;
+ let mut value = T::Native::ZERO;
+ let mut chunk = 0_u64;
+ let mut chunk_len = 0_usize;
+ let mut saw_point = false;
+ let mut int_kept = 0_usize;
+ let mut frac_kept = 0_usize;
+ let mut first_discarded_digit = None;
- // Overflow checks are not required if 10^(precision - 1) <= T::MAX holds.
- // Thus, if we validate the precision correctly, we can skip overflow
checks.
- while let Some((index, b)) = bs.next() {
+ let mut index = 0;
+ while let Some(&b) = mantissa.get(index) {
match b {
b'0'..=b'9' => {
- if digits == 0 && *b == b'0' {
- // Ignore leading zeros.
- continue;
- }
- digits += 1;
- result = result.mul_wrapping(base);
- result = result.add_wrapping(T::Native::usize_as((b - b'0') as
usize));
- }
- b'.' => {
- let point_index = index;
-
- for (_, b) in bs.by_ref() {
- if !b.is_ascii_digit() {
- if *b == b'e' || *b == b'E' {
- result = parse_e_notation::<T>(
- s,
- digits as u16,
- fractionals as i16,
- result,
- point_index + 1,
- precision as u16,
- scale as i16,
- )?;
-
- is_e_notation = true;
-
- break;
- }
- return Err(ArrowError::ParseError(format!(
- "can't parse the string value {s} to decimal"
- )));
- }
- if fractionals == scale {
- // We have processed all the digits that we need. All
that
- // is left is to validate that the rest of the string
contains
- // valid digits.
- continue;
+ let digit = b - b'0';
+ let (kept, keep) = if saw_point {
+ (&mut frac_kept, frac_keep)
+ } else {
+ (&mut int_kept, int_keep)
+ };
+ if *kept < keep {
+ *kept += 1;
+ // Cannot overflow: the chunk is folded into `value`
before it
+ // exceeds MAX_CHUNK_DIGITS digits, all of which fit in a
u64
+ chunk = chunk * 10 + digit as u64;
+ chunk_len += 1;
+ if chunk_len == MAX_CHUNK_DIGITS {
+ value = fold_decimal_chunk::<T>(value, chunk,
chunk_len, negative)?;
+ chunk = 0;
+ chunk_len = 0;
}
- fractionals += 1;
- digits += 1;
- result = result.mul_wrapping(base);
- result = result.add_wrapping(T::Native::usize_as((b -
b'0') as usize));
- }
-
- if is_e_notation {
- break;
+ } else {
+ first_discarded_digit.get_or_insert(digit);
}
}
- b'e' | b'E' => {
- result = parse_e_notation::<T>(
- s,
- digits as u16,
- fractionals as i16,
- result,
- index,
- precision as u16,
- scale as i16,
- )?;
-
- is_e_notation = true;
-
- break;
- }
- _ => {
- return Err(ArrowError::ParseError(format!(
- "can't parse the string value {s} to decimal"
- )));
- }
+ b'.' if !saw_point => saw_point = true,
+ b'e' | b'E' => return Err(MantissaError::Exponent(index)),
+ _ => return Err(MantissaError::InvalidFormat),
}
+ index += 1;
}
- if !is_e_notation {
- if fractionals < scale {
- let exp = scale - fractionals;
- if exp as u8 + digits > precision {
- return Err(ArrowError::ParseError(format!(
- "parse decimal overflow ({s})"
- )));
- }
- let mul = base.pow_wrapping(exp as _);
- result = result.mul_wrapping(mul);
- } else if digits > precision {
- return Err(ArrowError::ParseError(format!(
- "parse decimal overflow ({s})"
- )));
+ if chunk_len > 0 {
+ value = fold_decimal_chunk::<T>(value, chunk, chunk_len, negative)?;
+ }
+
+ if int_kept == 0 && frac_kept == 0 && first_discarded_digit.is_none() {
+ return Err(MantissaError::InvalidFormat);
+ }
+
+ // Scale the value up to the target scale. Skipped for zero, where
computing
+ // 10^missing could overflow the native type even though the result (zero)
+ // is always representable.
+ let missing = scale - frac_kept as i64;
+ if missing > 0 && !value.is_zero() {
+ value = value
+ .mul_checked(decimal_pow::<T>(missing)?)
+ .map_err(|_| MantissaError::Overflow)?;
+ }
+
+ round &= first_discarded_digit.is_some_and(|digit| digit >= 5);
+ if round {
+ value = if negative {
+ value.sub_checked(T::Native::ONE)
+ } else {
+ value.add_checked(T::Native::ONE)
+ }
+ .map_err(|_| MantissaError::Overflow)?;
+ }
+
+ Ok(value)
+}
+
+/// Parses the digits of an exponent (`[+|-] digits`), saturating at the bounds
+/// of `i64`; any exponent that large scales every non-zero mantissa out of
+/// range of every decimal type.
+fn parse_decimal_exponent(exponent: &[u8]) -> Result<i64, DecimalParseError> {
+ let (negative, digits) = split_sign(exponent);
+ if digits.is_empty() {
+ return Err(DecimalParseError::InvalidFormat);
+ }
+ let mut value = 0_i64;
+ for &b in digits {
+ if !b.is_ascii_digit() {
+ return Err(DecimalParseError::InvalidFormat);
}
+ value = value.saturating_mul(10).saturating_add((b - b'0') as i64);
}
+ Ok(if negative { -value } else { value })
+}
- Ok(if negative {
- result.neg_wrapping()
+/// Splits an optional leading sign from `bytes`, returning whether it is `-`
+/// and the bytes that follow it.
+#[inline]
+fn split_sign(bytes: &[u8]) -> (bool, &[u8]) {
+ match bytes.first() {
+ Some(b'-') => (true, &bytes[1..]),
+ Some(b'+') => (false, &bytes[1..]),
+ _ => (false, bytes),
+ }
+}
+
+/// Folds a chunk of up to [`MAX_CHUNK_DIGITS`] digits into `value`, producing
+/// `value * 10^chunk_len + chunk` (`chunk` is negated first when parsing a
+/// negative number).
+#[inline]
+fn fold_decimal_chunk<T: DecimalType>(
+ value: T::Native,
+ chunk: u64,
+ chunk_len: usize,
+ negative: bool,
+) -> Result<T::Native, DecimalParseError> {
+ let chunk = decimal_chunk_to_native::<T>(chunk, negative)?;
+
+ // When `value` is zero the multiply would be a no-op; skipping it avoids
+ // computing 10^chunk_len, which can overflow a narrow native type even
+ // though the result (the chunk itself) is representable.
+ if value.is_zero() {
+ return Ok(chunk);
+ }
+
+ value
+ .mul_checked(decimal_pow::<T>(chunk_len as i64)?)
+ .map_err(|_| DecimalParseError::Overflow)?
+ .add_checked(chunk)
+ .map_err(|_| DecimalParseError::Overflow)
+}
+
+/// Converts a chunk of at most [`MAX_CHUNK_DIGITS`] digits to the native type,
+/// negated if `negative`.
+#[inline]
+fn decimal_chunk_to_native<T: DecimalType>(
+ chunk: u64,
+ negative: bool,
+) -> Result<T::Native, DecimalParseError> {
+ // Every native type can represent +/- 10^9, so a chunk below that converts
+ // losslessly through usize on every target. So does any chunk when the
+ // native type holds MAX_CHUNK_DIGITS digits and usize holds a u64.
+ const HALF: u64 = 1_000_000_000;
+ if chunk < HALF || (T::MAX_PRECISION as usize >= MAX_CHUNK_DIGITS &&
usize::BITS >= 64) {
+ let chunk = T::Native::usize_as(chunk as usize);
+ // `ZERO.sub_wrapping` rather than `neg_wrapping`: the latter compiles
+ // to measurably slower code for i256 (~10% on casting strings to
+ // Decimal256)
+ return Ok(if negative {
+ T::Native::ZERO.sub_wrapping(chunk)
+ } else {
+ chunk
+ });
+ }
+ // Otherwise narrow the chunk in two halves that are each below 10^9
+ let low = T::Native::usize_as((chunk % HALF) as usize);
+ let high = T::Native::usize_as((chunk / HALF) as usize)
+ .mul_checked(T::Native::usize_as(HALF as usize))
+ .map_err(|_| DecimalParseError::Overflow)?;
+ // Negate before combining so that a chunk with the magnitude of the
+ // native type's minimum value (e.g. "2147483648" for Decimal32) remains
+ // representable
+ if negative {
+ T::Native::ZERO
+ .sub_checked(high)
+ .map_err(|_| DecimalParseError::Overflow)?
+ .sub_checked(low)
+ .map_err(|_| DecimalParseError::Overflow)
} else {
- result
- })
+ high.add_checked(low)
+ .map_err(|_| DecimalParseError::Overflow)
+ }
+}
+
+/// Returns `10^exp` as a `T::Native`, or an overflow error if the result does
+/// not fit in the native type.
+#[inline]
+fn decimal_pow<T: DecimalType>(exp: i64) -> Result<T::Native,
DecimalParseError> {
+ // T::MAX_FOR_EACH_PRECISION[k] holds 10^k - 1, so adding one yields 10^k
+ // without computing a power at runtime. Exponents beyond the table always
+ // overflow: the native type cannot hold 10^(MAX_PRECISION + 1).
+ usize::try_from(exp)
+ .ok()
+ .and_then(|exp| T::MAX_FOR_EACH_PRECISION.get(exp))
+ .map(|max| max.add_wrapping(T::Native::ONE))
+ .ok_or(DecimalParseError::Overflow)
}
/// Parse human-readable interval string to Arrow [IntervalYearMonthType]
@@ -2658,6 +2761,9 @@ mod tests {
("4749.3e+5", "474930000", 1),
("0E-8", "0", 10),
("0E+6", "0", 10),
+ ("0e0", "0", 10),
+ ("-0e0", "0", 10),
+ ("00e48", "0", 10),
("1E-8", "0.00000001", 10),
("12E+6", "12000000", 10),
("12E-6", "0.000012", 10),
@@ -2669,14 +2775,17 @@ mod tests {
("000001.1034567002e0", "000001.1034567002", 3),
("1.234e16", "12340000000000000", 0),
("123.4e16", "1234000000000000000", 0),
+ ("15e-1", "1.5", 0),
+ ("1.25e1", "12.5", 0),
+ ("1.5e-1", "0.15", 1),
];
for (e, d, scale) in e_notation_tests {
let result_128_e = parse_decimal::<Decimal128Type>(e, 20, scale);
let result_128_d = parse_decimal::<Decimal128Type>(d, 20, scale);
- assert_eq!(result_128_e.unwrap(), result_128_d.unwrap());
+ assert_eq!(result_128_e.unwrap(), result_128_d.unwrap(), "{e} vs
{d}");
let result_256_e = parse_decimal::<Decimal256Type>(e, 20, scale);
let result_256_d = parse_decimal::<Decimal256Type>(d, 20, scale);
- assert_eq!(result_256_e.unwrap(), result_256_d.unwrap());
+ assert_eq!(result_256_e.unwrap(), result_256_d.unwrap(), "{e} vs
{d}");
}
let can_not_parse_tests = [
"123,123",
@@ -2686,6 +2795,11 @@ mod tests {
"+",
"-",
"e",
+ "e5",
+ "-.",
+ "+e-11",
+ "-.E+3",
+ ".e5",
"1.3e+e3",
"5.6714ee-2",
"4.11ee-+4",
@@ -2696,16 +2810,26 @@ mod tests {
"1e",
"1e+",
"1e-",
+ "1e5e5",
+ "1 000",
+ "1_000",
+ "- 1",
+ "1.5 x",
+ "0x10",
+ "NaN",
+ "inf",
+ "\u{661}\u{662}",
+ "\u{ff11}",
];
for s in can_not_parse_tests {
let result_128 = parse_decimal::<Decimal128Type>(s, 20, 3);
assert_eq!(
- format!("Parser error: can't parse the string value {s} to
decimal"),
+ format!("Parser error: Invalid decimal format: {s:?}"),
result_128.unwrap_err().to_string()
);
let result_256 = parse_decimal::<Decimal256Type>(s, 20, 3);
assert_eq!(
- format!("Parser error: can't parse the string value {s} to
decimal"),
+ format!("Parser error: Invalid decimal format: {s:?}"),
result_256.unwrap_err().to_string()
);
}
@@ -2721,25 +2845,20 @@ mod tests {
("1234560000000", 0),
("12345678900.0", 0),
("1.23456e12", 0),
+ ("9999999.9995", 3),
+ ("1e99999", 0),
+ ("1e40", 0),
];
for (s, scale) in overflow_parse_tests {
let result_128 = parse_decimal::<Decimal128Type>(s, 10, scale);
- let expected_128 = "Parser error: parse decimal overflow";
- let actual_128 = result_128.unwrap_err().to_string();
-
- assert!(
- actual_128.contains(expected_128),
- "actual: '{actual_128}', expected: '{expected_128}'"
- );
+ let expected_128 =
+ format!("Parser error: {s:?} does not fit in Decimal128(10,
{scale})");
+ assert_eq!(result_128.unwrap_err().to_string(), expected_128);
let result_256 = parse_decimal::<Decimal256Type>(s, 10, scale);
- let expected_256 = "Parser error: parse decimal overflow";
- let actual_256 = result_256.unwrap_err().to_string();
-
- assert!(
- actual_256.contains(expected_256),
- "actual: '{actual_256}', expected: '{expected_256}'"
- );
+ let expected_256 =
+ format!("Parser error: {s:?} does not fit in Decimal256(10,
{scale})");
+ assert_eq!(result_256.unwrap_err().to_string(), expected_256);
}
let edge_tests_128 = [
@@ -2777,11 +2896,25 @@ mod tests {
("-1e3", -1000000000i128, 6),
("+1e3", 1000000000i128, 6),
("-1e31", -10000000000000000000000000000000000000i128, 6),
+ // More digits than an i128 can hold, but a small value
+ ("10000000000000000000000000000000000000000e-39", 10i128, 0),
+ // Digits beyond the scale round; here the result still fits
+ (
+ "99999999999999999999999999999999999994e-1",
+ 9999999999999999999999999999999999999i128,
+ 0,
+ ),
];
for (s, i, scale) in edge_tests_128 {
let result_128 = parse_decimal::<Decimal128Type>(s, 38, scale);
- assert_eq!(i, result_128.unwrap());
+ assert_eq!(i, result_128.unwrap(), "{s}");
}
+ // Rounding carries into a 39th digit, which does not fit
+ assert!(
+
parse_decimal::<Decimal128Type>("999999999999999999999999999999999999999e-1",
38, 0)
+ .is_err()
+ );
+
let edge_tests_256 = [
(
"9999999999999999999999999999999999999999999999999999999999999999999999999999",
@@ -2846,10 +2979,13 @@ mod tests {
("1.23", 1, 3),
("1.000", 1, 3),
("1.123", 1, 3),
+ ("1.5", 2, 3),
+ ("1.9", 2, 3),
("123.0", 123, 3),
("123.4", 123, 3),
("123.00", 123, 3),
("123.45", 123, 3),
+ ("123.5", 124, 3),
("123.000000000000000000004", 123, 3),
("0.123e2", 12, 3),
("0.123e4", 1230, 10),
@@ -2864,19 +3000,455 @@ mod tests {
];
for (s, i, precision) in zero_scale_tests {
let result_128 = parse_decimal::<Decimal128Type>(s, precision,
0).unwrap();
- assert_eq!(i, result_128);
+ assert_eq!(i, result_128, "{s}");
}
let can_not_parse_zero_scale = [".", "blag", "", "+", "-", "e"];
for s in can_not_parse_zero_scale {
let result_128 = parse_decimal::<Decimal128Type>(s, 5, 0);
assert_eq!(
- format!("Parser error: can't parse the string value {s} to
decimal"),
+ format!("Parser error: Invalid decimal format: {s:?}"),
result_128.unwrap_err().to_string(),
);
}
}
+ #[test]
+ fn test_parse_decimal_rounds_half_away_from_zero() {
+ let tests = [
+ ("1.234", 2, 123),
+ ("1.235", 2, 124),
+ ("1.2350000", 2, 124),
+ ("1.2349999", 2, 123),
+ ("-1.234", 2, -123),
+ ("-1.235", 2, -124),
+ ("-0.004", 2, 0),
+ ("-0.005", 2, -1),
+ (".5", 0, 1),
+ ("-.5", 0, -1),
+ ("0.5", 0, 1),
+ ("1.5", 0, 2),
+ ("2.5", 0, 3),
+ ("-2.5", 0, -3),
+ ("1.99", 1, 20),
+ ("0.995", 2, 100),
+ ("9.99", 1, 100),
+ ("123.4567891", 5, 12345679),
+ ("123.45", 0, 123),
+ ("0.0000123", 3, 0),
+ ("12.", 2, 1200),
+ (".12", 2, 12),
+ ("+.12", 2, 12),
+ ("-.12", 2, -12),
+ ];
+ for (s, scale, expected) in tests {
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(s, 38, scale).unwrap(),
+ expected,
+ "{s} at scale {scale}"
+ );
+ assert_eq!(
+ parse_decimal::<Decimal256Type>(s, 76, scale).unwrap(),
+ i256::from_i128(expected),
+ "{s} at scale {scale}"
+ );
+ }
+ }
+
+ #[test]
+ fn test_parse_decimal_rounding_overflow() {
+ // Rounding up can push the value past the precision ...
+ assert!(parse_decimal::<Decimal128Type>("99999.5", 5, 0).is_err());
+ assert!(parse_decimal::<Decimal128Type>("9.995", 3, 2).is_err());
+ assert_eq!(parse_decimal::<Decimal128Type>("9.994", 3, 2).unwrap(),
999);
+ assert_eq!(parse_decimal::<Decimal128Type>("0.995", 3, 2).unwrap(),
100);
+
+ // ... or past the native type itself
+ assert_eq!(
+ parse_decimal_native::<Decimal32Type>("2147483647.5", 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal32Type>("-2147483648.5", 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal128Type>(&format!("{}.5",
i128::MAX), 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal128Type>(&format!("{}.5",
i128::MIN), 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal256Type>(&format!("{}.5",
i256::MAX), 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal256Type>(&format!("{}.5",
i256::MIN), 0),
+ Err(DecimalParseError::Overflow)
+ );
+ }
+
+ #[test]
+ fn test_parse_decimal_native_full_range() {
+ // The native range exceeds the largest precision; the precision check
+ // is the caller's responsibility
+ assert_eq!(
+ parse_decimal_native::<Decimal32Type>("-2147483648", 0),
+ Ok(i32::MIN)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal32Type>("2147483648", 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal64Type>("-9223372036854775808", 0),
+ Ok(i64::MIN)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal64Type>("9223372036854775808", 0),
+ Err(DecimalParseError::Overflow)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal128Type>(&i128::MAX.to_string(), 0),
+ Ok(i128::MAX)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal128Type>(&i128::MIN.to_string(), 0),
+ Ok(i128::MIN)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal256Type>(&i256::MAX.to_string(), 0),
+ Ok(i256::MAX)
+ );
+ assert_eq!(
+ parse_decimal_native::<Decimal256Type>(&i256::MIN.to_string(), 0),
+ Ok(i256::MIN)
+ );
+ // The unscaled value (integer digits scaled by 10^21) far exceeds the
+ // i256 range, so this must report overflow rather than wrapping to an
+ // arbitrary (possibly in-range) value
+ let input = format!("{}.12345678901234567890123", "7".repeat(71));
+ assert_eq!(
+ parse_decimal_native::<Decimal256Type>(&input, 21),
+ Err(DecimalParseError::Overflow)
+ );
+
+ assert!(parse_decimal::<Decimal128Type>(&i128::MAX.to_string(), 38,
0).is_err());
+ assert!(parse_decimal::<Decimal32Type>("-2147483648", 9, 0).is_err());
+ }
+
+ #[test]
+ fn test_parse_decimal_integer_widths() {
+ assert_eq!(
+ parse_decimal::<Decimal32Type>("123.45", 9, 2).unwrap(),
+ 12_345_i32
+ );
+ assert_eq!(
+ parse_decimal::<Decimal32Type>("-9999999.994", 9, 2).unwrap(),
+ -999_999_999_i32
+ );
+ assert!(parse_decimal::<Decimal32Type>("9999999.995", 9, 2).is_err());
+ assert!(parse_decimal::<Decimal32Type>("-9999999.995", 9, 2).is_err());
+ assert_eq!(
+ parse_decimal::<Decimal64Type>("123.45", 18, 2).unwrap(),
+ 12_345_i64
+ );
+ assert_eq!(
+ parse_decimal::<Decimal64Type>("9999999999999999.99", 18,
2).unwrap(),
+ 999_999_999_999_999_999_i64
+ );
+ assert!(parse_decimal::<Decimal64Type>("10000000000000000.00", 18,
2).is_err());
+ // Fractional parts longer than any native integer type parse fine;
+ // digits beyond the scale only matter for rounding
+ assert_eq!(
+ parse_decimal::<Decimal64Type>(&format!(".{}", "5".repeat(100)),
18, 4).unwrap(),
+ 5_556_i64
+ );
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(&format!(".{}", "1".repeat(100)),
38, 4).unwrap(),
+ 1_111_i128
+ );
+ }
+
+ #[test]
+ fn test_parse_decimal_exponent() {
+ let tests = [
+ ("1e2", 0, 100),
+ ("1E2", 0, 100),
+ ("1e+2", 0, 100),
+ ("1e+02", 0, 100),
+ ("1.5e2", 0, 150),
+ ("1.5e2", 2, 15000),
+ ("1.5e-1", 1, 2),
+ ("15e-1", 0, 2),
+ ("1e-2", 1, 0),
+ ("1e-3", 2, 0),
+ ("0e0", 2, 0),
+ ("-0e0", 2, 0),
+ ("0E5", 2, 0),
+ ("0e99999", 2, 0),
+ ("00e48", 8, 0),
+ ("+00.0E+41", 12, 0),
+ ("1.25e1", 0, 13),
+ ("1e-99999", 2, 0),
+ ("1.5e-400", 2, 0),
+ ("123456789e-9", 9, 123456789),
+ ("0.000000001e9", 0, 1),
+ ("5e-1", 0, 1),
+ ("4e-1", 0, 0),
+ ("-5e-1", 0, -1),
+ ];
+ for (s, scale, expected) in tests {
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(s, 38, scale).unwrap(),
+ expected,
+ "{s} at scale {scale}"
+ );
+ assert_eq!(
+ parse_decimal::<Decimal32Type>(s, 9, scale).unwrap(),
+ expected as i32,
+ "{s} at scale {scale}"
+ );
+ }
+
+ // Exponents shift digits across the decimal point without losing any
+ assert_eq!(
+
parse_decimal::<Decimal32Type>("4825037936439135476.2609835314269495255615E-14",
9, 4)
+ .unwrap(),
+ 482503794
+ );
+ assert_eq!(
+ parse_decimal::<Decimal32Type>(
+
"+18232335063972188138031550982650807591758238.0724251287782783777442440E-58",
+ 1,
+ 0
+ )
+ .unwrap(),
+ 0
+ );
+ assert_eq!(
+
parse_decimal::<Decimal128Type>("4825037936439135476.2609835314269495255615E-14",
9, 1)
+ .unwrap(),
+ 482504
+ );
+ assert!(
+
parse_decimal::<Decimal128Type>("4825037936439135476.2609835314269495255615E-14",
5, 1)
+ .is_err()
+ );
+ // Absurdly long exponents saturate rather than wrap
+ assert!(parse_decimal::<Decimal128Type>(&format!("1e{}",
"9".repeat(30)), 38, 0).is_err());
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(&format!("1e-{}", "9".repeat(30)),
38, 0).unwrap(),
+ 0
+ );
+ }
+
+ #[test]
+ fn test_parse_decimal_negative_scale() {
+ let tests = [
+ ("1234.5", -2, 12),
+ ("150", -2, 2),
+ ("149", -2, 1),
+ ("-150", -2, -2),
+ ("-149", -2, -1),
+ ("50", -2, 1),
+ ("49", -2, 0),
+ ("5", -1, 1),
+ ("4", -1, 0),
+ ("0.5", -1, 0),
+ ("5.9", -1, 1),
+ ("1e5", -2, 1000),
+ ("1.5e5", -2, 1500),
+ ("0.9e2", -1, 9),
+ (".5e3", -2, 5),
+ ("12345", -5, 0),
+ ("12345", -4, 1),
+ ("000123456", -3, 123),
+ ("0", -5, 0),
+ ("-0.0", -5, 0),
+ ];
+ for (s, scale, expected) in tests {
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(s, 38, scale).unwrap(),
+ expected,
+ "{s} at scale {scale}"
+ );
+ assert_eq!(
+ parse_decimal::<Decimal32Type>(s, 9, scale).unwrap(),
+ expected as i32,
+ "{s} at scale {scale}"
+ );
+ assert_eq!(
+ parse_decimal::<Decimal256Type>(s, 76, scale).unwrap(),
+ i256::from_i128(expected),
+ "{s} at scale {scale}"
+ );
+ }
+ // The integer part can be wider than the native type as long as the
+ // scaled value fits
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(&format!("1{}", "0".repeat(50)),
38, -40).unwrap(),
+ 10_000_000_000
+ );
+ assert_eq!(
+ parse_decimal::<Decimal32Type>("123456789012", 9, -5).unwrap(),
+ 1234568
+ );
+ assert!(parse_decimal::<Decimal32Type>("123456789012", 9,
-2).is_err());
+ }
+
+ #[test]
+ fn test_parse_decimal_whitespace_and_long_input() {
+ for s in [" 1.5", "1.5 ", " 1.5 ", "\t1.5\n", "\r\n1.5\x0c"] {
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(s, 38, 1).unwrap(),
+ 15,
+ "{s:?}"
+ );
+ }
+ // Only ASCII whitespace is trimmed, as for the other CSV parsers
+ assert!(parse_decimal::<Decimal128Type>("\u{a0}1.5", 38, 1).is_err());
+ assert!(parse_decimal::<Decimal128Type>("1.5\u{2003}", 38,
1).is_err());
+ assert!(parse_decimal::<Decimal128Type>(" ", 38, 1).is_err());
+
+ // Long inputs report overflow rather than wrapping or panicking
+ for s in [
+ "1".repeat(255),
+ "1".repeat(256),
+ "1".repeat(300),
+ format!("{}.5", "1".repeat(300)),
+ format!("1e{}", "9".repeat(300)),
+ ] {
+ let err = parse_decimal::<Decimal128Type>(&s, 38, 0).unwrap_err();
+ assert!(err.to_string().contains("does not fit"), "{err}");
+ }
+ // Long fractions only matter for rounding
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(&format!("0.{}", "0".repeat(200)),
38, 10).unwrap(),
+ 0
+ );
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(&format!("0.{}1",
"0".repeat(200)), 38, 10).unwrap(),
+ 0
+ );
+ assert_eq!(
+ parse_decimal::<Decimal128Type>(&format!("1.{}", "9".repeat(300)),
38, 2).unwrap(),
+ 200
+ );
+ // 10^scale overflows the native type, but zero is still representable
+ assert_eq!(parse_decimal::<Decimal32Type>("0", 9, 10).unwrap(), 0);
+ assert_eq!(parse_decimal::<Decimal32Type>("-0.0", 9, 10).unwrap(), 0);
+ assert_eq!(parse_decimal::<Decimal64Type>("0", 18, 20).unwrap(), 0);
+ assert_eq!(parse_decimal::<Decimal128Type>("0", 38, 40).unwrap(), 0);
+ assert!(parse_decimal::<Decimal32Type>("1", 9, 10).is_err());
+ }
+
+ #[test]
+ #[cfg_attr(miri, ignore)] // Takes too long under Miri (adds ~1 hour to CI)
+ fn test_parse_decimal_matches_bigint_reference() {
+ use num_bigint::BigInt;
+ use rand::rngs::StdRng;
+ use rand::{RngExt, SeedableRng};
+
+ /// Generates random decimal strings with a known exact value and
checks
+ /// that `parse_decimal` rounds them correctly or reports overflow
+ fn check<T: DecimalType>(rng: &mut StdRng, iterations: usize)
+ where
+ T::Native: std::fmt::Display,
+ {
+ let random_digits = |rng: &mut StdRng, len: usize| -> String {
+ (0..len)
+ .map(|_| char::from(b'0' + rng.random_range(0..10u8)))
+ .collect()
+ };
+ for _ in 0..iterations {
+ let sign = ["", "+", "-"][rng.random_range(0..3)];
+ let int_len = rng.random_range(0..=40);
+ let frac_len = if rng.random_bool(0.3) {
+ 0
+ } else {
+ rng.random_range(0..=40)
+ };
+ if int_len == 0 && frac_len == 0 {
+ continue;
+ }
+ let int = random_digits(rng, int_len);
+ let frac = random_digits(rng, frac_len);
+ let mut s = format!("{sign}{int}");
+ if frac_len > 0 || rng.random_bool(0.2) {
+ s.push('.');
+ s.push_str(&frac);
+ }
+ let exponent: i64 = if rng.random_bool(0.3) {
+ rng.random_range(-60..=60)
+ } else {
+ 0
+ };
+ if exponent != 0 || rng.random_bool(0.1) {
+ s.push(if rng.random_bool(0.5) { 'e' } else { 'E' });
+ if exponent >= 0 && rng.random_bool(0.5) {
+ s.push('+');
+ }
+ s.push_str(&exponent.to_string());
+ }
+ let precision = rng.random_range(1..=T::MAX_PRECISION);
+ let scale = rng.random_range(-10..=T::MAX_SCALE.min(precision
as i8));
+
+ // value = mantissa * 10^(exponent - frac_len), scaled by
10^scale
+ // and rounded half away from zero
+ let mantissa: BigInt = format!("{int}{frac}").parse().unwrap();
+ let shift = exponent - frac_len as i64 + scale as i64;
+ let mut expected = if shift >= 0 {
+ mantissa * BigInt::from(10).pow(shift as u32)
+ } else {
+ let divisor = BigInt::from(10).pow((-shift) as u32);
+ let quotient = &mantissa / &divisor;
+ if (&mantissa % &divisor) * 2 >= divisor {
+ quotient + 1
+ } else {
+ quotient
+ }
+ };
+ if sign == "-" {
+ expected = -expected;
+ }
+ let limit = BigInt::from(10).pow(precision as u32);
+ let fits = expected < limit && expected > -limit;
+
+ match (fits, parse_decimal::<T>(&s, precision, scale)) {
+ (true, Ok(actual)) => {
+ let actual: BigInt =
actual.to_string().parse().unwrap();
+ assert_eq!(
+ actual,
+ expected,
+ "{s:?} as {}({precision}, {scale})",
+ T::PREFIX
+ );
+ }
+ (false, Err(_)) => {}
+ (true, Err(e)) => {
+ panic!(
+ "{s:?} as {}({precision}, {scale}): expected
{expected}, got {e}",
+ T::PREFIX
+ )
+ }
+ (false, Ok(actual)) => panic!(
+ "{s:?} as {}({precision}, {scale}): expected overflow,
got {actual}",
+ T::PREFIX
+ ),
+ }
+ }
+ }
+
+ let mut rng = StdRng::seed_from_u64(0xDEC1_3A15);
+ check::<Decimal32Type>(&mut rng, 5_000);
+ check::<Decimal64Type>(&mut rng, 5_000);
+ check::<Decimal128Type>(&mut rng, 5_000);
+ check::<Decimal256Type>(&mut rng, 5_000);
+ }
+
#[test]
fn test_parse_empty() {
assert_eq!(Int32Type::parse(""), None);
diff --git a/arrow-csv/src/reader/mod.rs b/arrow-csv/src/reader/mod.rs
index 6f696482f1..3b7cce2dc3 100644
--- a/arrow-csv/src/reader/mod.rs
+++ b/arrow-csv/src/reader/mod.rs
@@ -1386,6 +1386,7 @@ mod tests {
use tempfile::NamedTempFile;
use arrow_array::cast::AsArray;
+ use arrow_cast::display::array_value_to_string;
#[test]
fn test_csv() {
@@ -1459,7 +1460,7 @@ mod tests {
assert_eq!("53.002666", lat.value_as_string(1));
assert_eq!("52.412811", lat.value_as_string(2));
assert_eq!("51.481583", lat.value_as_string(3));
- assert_eq!("12.123456", lat.value_as_string(4));
+ assert_eq!("12.123457", lat.value_as_string(4));
assert_eq!("50.760000", lat.value_as_string(5));
assert_eq!("0.123000", lat.value_as_string(6));
assert_eq!("123.000000", lat.value_as_string(7));
@@ -1484,6 +1485,61 @@ mod tests {
assert_eq!("0.290472", lng.value_as_string(9));
}
+ #[test]
+ fn test_csv_reader_decimal_parsing() {
+ // Rounding half away from zero, surrounding whitespace, exponent
+ // notation and negative scales are all accepted
+ let data = " 1.995
,1.5e2,1234.5,0e0\n-0.005,-1.5E-2,-150,1E+2\n123,+.5,5,-7\n";
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Decimal128(10, 2), false),
+ Field::new("b", DataType::Decimal64(18, 2), false),
+ Field::new("c", DataType::Decimal128(10, -2), false),
+ Field::new("d", DataType::Decimal32(9, 0), false),
+ ]));
+ let mut csv =
ReaderBuilder::new(schema).build(Cursor::new(data)).unwrap();
+ let batch = csv.next().unwrap().unwrap();
+ let column = |i: usize| {
+ (0..batch.num_rows())
+ .map(|row| array_value_to_string(batch.column(i),
row).unwrap())
+ .collect::<Vec<_>>()
+ };
+ assert_eq!(column(0), ["2.00", "-0.01", "123.00"]);
+ assert_eq!(column(1), ["150.00", "-0.02", "0.50"]);
+ assert_eq!(
+ batch.column(2).as_primitive::<Decimal128Type>().values(),
+ &[12, -2, 0]
+ );
+ assert_eq!(column(3), ["0", "100", "-7"]);
+
+ // Invalid and out-of-range values are errors, never panics
+ for (data, expected) in [
+ ("abc\n", "Invalid decimal format: \"abc\""),
+ ("1.2.3\n", "Invalid decimal format: \"1.2.3\""),
+ (
+ "123456789\n",
+ "\"123456789\" does not fit in Decimal128(5, 2)",
+ ),
+ ("1e99999\n", "does not fit in Decimal128(5, 2)"),
+ (
+ &format!("{}\n", "1".repeat(300)),
+ "does not fit in Decimal128(5, 2)",
+ ),
+ (
+ "4825037936439135476.2609835314269495255615E-14\n",
+ "does not fit in Decimal128(5, 2)",
+ ),
+ ] {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "a",
+ DataType::Decimal128(5, 2),
+ false,
+ )]));
+ let mut csv =
ReaderBuilder::new(schema).build(Cursor::new(data)).unwrap();
+ let err = csv.next().unwrap().unwrap_err().to_string();
+ assert!(err.contains(expected), "{data:?}: {err}");
+ }
+ }
+
#[test]
fn test_csv_reader_with_decimal_3264() {
let schema = Arc::new(Schema::new(vec![
@@ -1507,7 +1563,7 @@ mod tests {
assert_eq!("53.002666", lat.value_as_string(1));
assert_eq!("52.412811", lat.value_as_string(2));
assert_eq!("51.481583", lat.value_as_string(3));
- assert_eq!("12.123456", lat.value_as_string(4));
+ assert_eq!("12.123457", lat.value_as_string(4));
assert_eq!("50.760000", lat.value_as_string(5));
assert_eq!("0.123000", lat.value_as_string(6));
assert_eq!("123.000000", lat.value_as_string(7));
diff --git a/arrow-json/src/reader/mod.rs b/arrow-json/src/reader/mod.rs
index ad17402ff8..f562295e2d 100644
--- a/arrow-json/src/reader/mod.rs
+++ b/arrow-json/src/reader/mod.rs
@@ -1494,7 +1494,7 @@ mod tests {
assert!(col1.is_null(5));
assert_eq!(
col1.values(),
- &[100, 200, 204, 1103420, 0, 0].map(T::Native::usize_as)
+ &[100, 200, 205, 1103420, 0, 0].map(T::Native::usize_as)
);
let col2 = batches[0].column(1).as_primitive::<T>();
@@ -1514,10 +1514,100 @@ mod tests {
assert!(col3.is_null(5));
assert_eq!(
col3.values(),
- &[3830, 12345, 0, 0, 0, 0].map(T::Native::usize_as)
+ &[3830, 12346, 0, 0, 0, 0].map(T::Native::usize_as)
);
}
+ #[test]
+ fn test_decimal_number_formats() {
+ // Rounding half away from zero, exponent notation (a valid JSON
+ // number syntax), surrounding whitespace in strings and negative
+ // scales are all accepted
+ let buf = r#"
+ {"a": 0e0, "b": " 1.5 ", "c": 1234.5}
+ {"a": 1.5E2, "b": "-0.005", "c": -150}
+ {"a": 1e-3, "b": "1e2", "c": 1E2}
+ {"a": -0E+0, "b": "+.5", "c": 5}
+ "#;
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("a", DataType::Decimal128(10, 2), true),
+ Field::new("b", DataType::Decimal64(18, 2), true),
+ Field::new("c", DataType::Decimal128(10, -2), true),
+ ]));
+ let batches = do_read(buf, 1024, true, false, schema);
+ assert_eq!(batches.len(), 1);
+ assert_eq!(
+ batches[0]
+ .column(0)
+ .as_primitive::<Decimal128Type>()
+ .values(),
+ &[0, 15000, 0, 0]
+ );
+ assert_eq!(
+ batches[0]
+ .column(1)
+ .as_primitive::<Decimal64Type>()
+ .values(),
+ &[150, -1, 10000, 50]
+ );
+ assert_eq!(
+ batches[0]
+ .column(2)
+ .as_primitive::<Decimal128Type>()
+ .values(),
+ &[12, -2, 1, 0]
+ );
+
+ // Invalid and out-of-range values are errors (or nulls when type
+ // conflicts are ignored), never panics
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "a",
+ DataType::Decimal128(5, 2),
+ true,
+ )]));
+ let long = "1".repeat(300);
+ for (buf, expected) in [
+ (
+ r#"{"a": "abc"}"#.to_string(),
+ "Invalid decimal format: \"abc\"",
+ ),
+ (
+ r#"{"a": 123456789}"#.to_string(),
+ "does not fit in Decimal128(5, 2)",
+ ),
+ (
+ r#"{"a": 1e99999}"#.to_string(),
+ "does not fit in Decimal128(5, 2)",
+ ),
+ (
+ format!(r#"{{"a": {long}}}"#),
+ "does not fit in Decimal128(5, 2)",
+ ),
+ (
+ r#"{"a":
4825037936439135476.2609835314269495255615E-14}"#.to_string(),
+ "does not fit in Decimal128(5, 2)",
+ ),
+ ] {
+ let err = ReaderBuilder::new(schema.clone())
+ .build(Cursor::new(buf.as_bytes()))
+ .unwrap()
+ .next()
+ .unwrap()
+ .unwrap_err()
+ .to_string();
+ assert!(err.contains(expected), "{buf}: {err}");
+
+ let batch = ReaderBuilder::new(schema.clone())
+ .with_ignore_type_conflicts(true)
+ .build(Cursor::new(buf.as_bytes()))
+ .unwrap()
+ .next()
+ .unwrap()
+ .unwrap();
+ assert!(batch.column(0).is_null(0), "{buf}");
+ }
+ }
+
#[test]
fn test_decimals() {
test_decimal::<Decimal32Type>(DataType::Decimal32(8, 2));
diff --git a/parquet-variant-compute/src/type_conversion.rs
b/parquet-variant-compute/src/type_conversion.rs
index 7f09a9d4d8..b3d3b7094f 100644
--- a/parquet-variant-compute/src/type_conversion.rs
+++ b/parquet-variant-compute/src/type_conversion.rs
@@ -18,10 +18,11 @@
//! Module for transforming a typed arrow `Array` to `VariantArray`.
use arrow::array::ArrowNativeTypeOp;
+use arrow::compute::kernels::cast_utils::parse_decimal;
use arrow::compute::{
CastOptions, DecimalCast, cast_num_to_bool,
cast_single_string_to_boolean_default, num_cast,
- parse_string_to_decimal_native, rescale_decimal, single_bool_to_numeric,
- single_decimal_to_float_lossy, single_float_to_decimal,
+ rescale_decimal, single_bool_to_numeric, single_decimal_to_float_lossy,
+ single_float_to_decimal,
};
use arrow::datatypes::{
self, ArrowPrimitiveType, ArrowTimestampType, Decimal32Type,
Decimal64Type, Decimal128Type,
@@ -369,7 +370,7 @@ impl_timestamp_from_variant!(
/// - Decimal variants (`Decimal4/8/16`) use their embedded precision and scale
///
/// The value is rescaled to (`precision`, `scale`) using `rescale_decimal`
for integers,
-/// `single_float_to_decimal` for floats, and `parse_string_to_decimal_native`
for strings.
+/// `single_float_to_decimal` for floats, and `parse_decimal` for strings.
/// returns `None` if it cannot fit the requested precision.
pub(crate) fn variant_to_unscaled_decimal<O>(
variant: &Variant<'_, '_>,
@@ -413,12 +414,8 @@ where
),
Variant::Float(f) => single_float_to_decimal::<O>(<f64 as
From<f32>>::from(*f), mul),
Variant::Double(f) => single_float_to_decimal::<O>(*f, mul),
- // arrow-cast only support cast string to decimal with scale >=0 for
now
- // Please see `cast_string_to_decimal` in
arrow-cast/src/cast/decimal.rs for more detail
- Variant::String(v) if scale >= 0 =>
parse_string_to_decimal_native::<O>(v, scale as _).ok(),
- Variant::ShortString(v) if scale >= 0 => {
- parse_string_to_decimal_native::<O>(v, scale as _).ok()
- }
+ Variant::String(v) => parse_decimal::<O>(v, precision, scale).ok(),
+ Variant::ShortString(v) => parse_decimal::<O>(v, precision,
scale).ok(),
Variant::Decimal4(d) => rescale_decimal::<Decimal32Type, O>(
d.integer(),
VariantDecimal4::MAX_PRECISION,
