Dustin Smith created SPARK-60845:
------------------------------------
Summary: Avoid BigDecimal round trips for wide decimals that fit
in a long in UnsafeArrayData
Key: SPARK-60845
URL: https://issues.apache.org/jira/browse/SPARK-60845
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Dustin Smith
SPARK-59805 adds a fast path for {{DECIMAL}} values with precision > 18 whose
unscaled value fits in a {{long}} in {{UnsafeRow.getDecimal}},
{{UnsafeRow.setDecimal}} and {{UnsafeRowWriter.write}}: they are decoded from
and encoded to the row's big-endian two's-complement bytes directly, without
the {{byte[]}}, {{BigInteger}} and {{BigDecimal}} round trips, and read back as
compact (long-backed) {{Decimal}}s.
Arrays and maps of wide decimals use the same byte encoding but still take the
slow path:
* {{UnsafeArrayData.getDecimal}} allocates a {{byte[]}}, a {{BigInteger}} and
two {{BigDecimal}}s per element and returns a BigDecimal-backed {{Decimal}}.
* {{UnsafeArrayWriter.write(int, Decimal, int, int)}} calls
{{toJavaBigDecimal().unscaledValue().toByteArray()}} per element.
Proposed change: apply the same fast path to {{UnsafeArrayData.getDecimal}} and
{{UnsafeArrayWriter.write}}, reusing the encoder from SPARK-59805, so the array
format stays byte-identical. This mostly helps functions and aggregates over
{{array<decimal(p, s)>}} / {{map<..., decimal(p, s)>}} with p > 18, such as
{{collect_list}}, higher-order functions like {{aggregate}}, and {{explode}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
