Hi all, Thank you for your comments, here is where we land with the revised DECFLOAT SPIP: - There is a single arithmetic path in DECFLOAT v1 proposal - the pure Java implementation. Any native kernels are completely out of scope for the purpose of this DECFLOAT SPIP. - Regarding the pure-Java in-tree implementation, of course the ported arithmetic library will land together with the actual downstream consumer code, Serge's PR was just a POC. - Persistence is conditional on the logical type in Parquet, and Spark will not ship any private encoding in the meantime; but, in-memory engine representation is independent. - There are no more open items regarding the public API surface. For Arrow we propose an extension type over fixed-width binary, and a Spark-owned value class for the JVM type. - To address the incubation concern, the SPIP now defines measurable exit criteria for enabling the type by default and provides better estimates against the larger scope. - The notes regarding coercion, AtomicType, and ordering, are all corrected. DECFLOAT is a FractionalType, ordering will follow DOUBLE's conventions, and DECFLOAT(34) is widest.
This should resolve all SPIP contracts, whereas the exact implementation details and deeper technical discussions can be carried out further during code review in appropriate PRs. Best, Uroš On 2026/09/01 16:07:31 Uroš Bojanić wrote: > Hi all, > > In accordance with concerns raised in the first SPIP vote for DECFLOAT > (https://lists.apache.org/thread/nozkdbsc7oy69j581qtp9v69wtzv4tdl), please > note that the SPIP proposal has been revised and updated accordingly: > https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E. > > In short, we agree to move away from the initial JNI / libbid proposal. The > new proposal is to use a custom Java native IEEE 754 compliant port. > > Given that this DECFLOAT proposal is still actively evolving, we can keep > this DISCUSS thread open to any additional feedback and substantive concerns > about this SPIP to add the DECFLOAT data type! > > Best, > Uroš > > On 2026/08/18 16:00:13 Uroš Bojanić wrote: > > Hi all, > > > > I would like to start a discussion on the SPIP to add a new Spark SQL data > > type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 > > floating-point decimals with per-value exponents. > > > > JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820 > > > > Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type > > (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the > > gap between DECIMAL, which caps at precision 38 with a fixed per-column > > scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact: > > each value keeps its own exponent, arithmetic is decimal, and it supports > > signed zero, Inf, and NaN. It is additive, round-trips through a proposed > > Parquet logical type for cross-engine interop, and would ship config-gated > > during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior > > unchanged. > > > > Additional information is available in the SPIP document: > > https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E > > > > Please provide your feedback on the approach, scope, and the proposal > > described in the document. > > > > Thank you! > > > > Best, > > Uroš > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
