DB Tsai created SPARK-60696:
-------------------------------
Summary: Accept scientific notation Decimal input in Spark Connect
Rust
Key: SPARK-60696
URL: https://issues.apache.org/jira/browse/SPARK-60696
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Python Decimal.__str__ emits scientific notation for ordinary finite values,
including Decimal('1E+3') and Decimal('1E-7'). createDataFrame([(value,)],
StructType([StructField('v',DecimalType(18,7))])) currently fails with invalid
decimal: invalid digit found in string for both; Decimal('123.45') succeeds.
Reproduced on current master plus combined client improvements. Core
decimal_str_to_unscaled splits only decimal point and parses exponent text as
integer. Implement checked scientific notation scaling, retaining decimal
precision/scale semantics, overflow and invalid-input handling. Validate exact
Arrow IPC cells against Python/Arrow reference for positive/negative exponents,
zeros, signs, bounds, rounding and errors. Preserve fixed-point throughput; no
JVM needed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
