dbtsai opened a new pull request, #132:
URL: https://github.com/apache/spark-connect-rust/pull/132
`createDataFrame([(Decimal("1E+3"),)], DecimalType(...))` currently fails
while parsing the exponent as an integer digit. Parse scientific Decimal
strings with checked scaling so exactly representable values, including `1E-7`
at scale 7, produce the correct Arrow Decimal128 cells.
Scientific inputs that lose fractional digits, exceed the declared
precision, or use a negative target scale are rejected. This keeps the newly
accepted inputs exact while leaving the existing fixed-point conversion policy
unchanged. The parser does not allocate or iterate in proportion to an exponent.
Validation: 396 Rust library tests; all integration targets compiled with
and without `wasm-udf`; 298 tests in the existing CI-selected Python offline
suite; 24 new candidate integration cases and 12 independent reference
tuple/pandas IPC checks against frozen Python/PyArrow, including physical
Decimal128 cells; baseline regression sensitivity; formatting and diff checks.
No JVM or live Spark server was used. Clippy is unavailable in the offline
toolchain.
The unchanged frozen benchmark harness was run with five fresh samples per
client and case for Decimal tuple/pandas inputs and nine guards. All 165
samples passed. Fixed-point Decimal tuple/pandas construction medians were
23.01 → 21.67 ms and 25.88 → 24.34 ms; all nine guard ranges overlap. These
benchmarks use fixed-point values, while scientific correctness is covered by
the independent IPC tests. Existing fixed-point rounding/precision limitations
are deliberately unchanged.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
