neilconway opened a new pull request, #11002:
URL: https://github.com/apache/arrow-rs/pull/11002
# Which issue does this PR close?
- N/A
# Rationale for this change
Formatting a decimal value allocated two heap strings: the unscaled value
was converted with `to_string`, and `format_decimal_str` then built a second
String with the decimal point inserted.
arrow-data gains three functions that share one implementation of the
formatting rules:
- `write_decimal_str` (private) takes the digits of an unscaled value, with
an optional sign, and writes them to a fmt::Write with the decimal point
inserted `scale` digits from the right, adding leading zeros as needed.
- `write_decimal` is for callers that have an output to write to, such as
`ArrayFormatter`: it formats the native value into a stack buffer of digits and
passes them to write_decimal_str.
- `format_decimal` is for callers that need an owned String, such as
`DecimalType::format_decimal`: it formats the digits the same way, then
allocates a String of the required capacity and writes into it through
`write_decimal_str`.
`write_decimal` and `format_decimal` accept the native value of a decimal
type, i32, i64, i128 or i256, named by the new sealed `DecimalNativeType`
trait. `DecimalType` already names these types, but it lives in arrow-array,
which depends on arrow-data, so it cannot be used here.
The existing entry points are now implemented on top of write_decimal and
format_decimal. This reduces the number of heap allocations in `ArrayFormatter`
from 2 -> 0 (which improves performance writing CSV and JSON output), and from
2 -> 1 for `PrimitiveArray::value_as_string`.
format_decimal benchmarks, M4 Max:
case before after change
decimal32 (9, 2) 9 digits 593.45 220.13 -62.9%
decimal64 (18, 6) 18 digits 726.44 243.84 -66.4%
decimal128 (10, 2) 1 digit 486.70 190.16 -60.9%
decimal128 (10, 2) 5 digits 480.75 209.84 -56.4%
decimal128 (38, 10) 38 digits 776.75 303.65 -60.9%
decimal256 (76, 10) 38 digits 1751.70 1070.30 -38.9%
decimal256 (76, 10) 76 digits 2097.40 1507.80 -28.1%
The relative improvement for decimal256 is smaller because that case had
additional overhead; that has been addressed in a concurrent PR (#11000).
# What changes are included in this PR?
See above.
# Are these changes tested?
Yes; existing tests pass, new test added.
# Are there any user-facing changes?
No. Decimal output format is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
