dwsmith1983 opened a new pull request, #59273:
URL: https://github.com/apache/spark/pull/59273
### What changes were proposed in this pull request?
`ExprUtils.getDecimalParser` strips grouping commas with `s.replaceAll(",",
"")` for the default `en-US` locale. This PR uses `s.replace(",", "")` instead.
The pattern is a literal comma, so the result is the same, but `replace` does
not compile a regex.
### Why are the changes needed?
Performance. `String.replaceAll` compiles a `java.util.regex.Pattern` on
every call, and this parser runs once per decimal value in the CSV reader
(`UnivocityParser`), the JSON reader (decimals given as strings), the XML
reader and JSON/XML schema inference.
On a 2M row CSV file with 5 `DECIMAL(18,2)` columns (local[1], JDK 17, best
of 3):
| | Before | After |
|---|---|---|
| One parse (`new BigDecimal(s.replaceAll/replace(",", ""))`) | 85.4 ns |
14.8 ns |
| Read the file with the DECIMAL schema | 1638 ms | 842 ms |
| Read the same file with the columns as STRING (reference) | 628 ms | 671
ms |
The DECIMAL read is 1.95x faster. Decimal parsing used to add about 1 s over
reading the columns as strings and now adds about 0.17 s.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests already parse grouped values such as `1,000.01` and
`158,058,049.001` through this code with the `en-US` locale. They pass:
`UnivocityParserSuite`, `CSVInferSchemaSuite`, `JsonInferSchemaSuite`,
`JsonExpressionsSuite`, `XmlExpressionsSuite`, and the decimal tests in
`CSVv1Suite`, `JsonV1Suite` and `XmlSuite`.
The numbers above come from a local benchmark run before and after the
change on the same commit.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
