Hi Thomas, Thank you for laying out the requirements explicitly. We agree that requirements should come before selecting a physical representation. There is substantial agreement, but we see several requirements differently.
For context on engine integration, we have also published a draft Spark Project Improvement Proposal (SPIP): Add the DECFLOAT data type (https://docs.google.com/document/d/1mTSOoCvX7yYMSa6CpI6OPS-MQZocPxUtrrFOEiZjCPk/edit). The SPIP covers the proposed Spark SQL type and execution semantics, while this discussion focuses on the portable Parquet representation. Before choosing BID or a new 18-byte format, we suggest first agreeing on the following: - Precision: We agree that the format must losslessly support 38-digit values. Your proposal targets exactly 38 digits, while we think the format should provide a defined path to both narrower and wider precisions. - Range: We agree that the format should cover at least the complete finite decimal128 range. - Cohorts: Your proposal requires one canonical representation per numerical value. We think the original coefficient, exponent, and quantum should round-trip. - Special values: We agree on infinity and NaN. We additionally think engines supporting signed zero, signaling NaNs, and NaN payloads should be able to round-trip them. - Hashing and equality: We agree that numerically equal values must behave consistently in equality and hash-based operations. We do not think that requires canonicalizing the stored representation. - Performance: We take the reported results seriously, but think the candidate representations should be compared using equivalent semantics, public benchmarks, and independent implementations. - Validation: Correctness and interoperability should be validated using public test vectors, separately from performance evaluation. The sections below provide the rationale for each position. Precision and range --------------------------- We agree that a Parquet decimal-floating-point type must losslessly represent every numerical value in the DECIMAL(38,s) domain. We do not think the format should necessarily stop at exactly 38 digits, however. Precision 38 is an important interoperability boundary established by an original crop of DBMS, which does not establish that present or future requirements stop at 38 digits. Many open source or major newer DBMS have higher limits: - PostgreSQL Type: NUMERIC unconstrained Max precision: 131,072 before decimal / 16,383 after; declared NUMERIC(p,s) up to p=1000 Source: https://www.postgresql.org/docs/current/datatype-numeric.html - MySQL / MariaDB / SingleStore Type: DECIMAL Max precision: 65 Source: https://dev.mysql.com/doc/refman/8.0/en/fixed-point-types.html - BigQuery Type: BIGNUMERIC Max precision: ~76.8 (scale 38) Source: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types - ClickHouse Type: Decimal256 Max precision: 76 Source: https://clickhouse.com/docs/sql-reference/data-types/decimal - Oracle Type: unconstrained NUMBER Max precision: ~40 (parameterized 38) Source: https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlqr/Data-Types.html - Trino ≥480 Type: NUMBER Max precision: >= 50 Source: https://trino.io/docs/current/language/types.html#number While we can opine on the value of precision past a certain limit, we cannot pass judgement on whether the cut-off is 34, 38, 43 or something higher. Therefore Parquet should avoid making any such precision a permanent semantic ceiling. Faithful round-tripping and cohorts ---------------------------------------------- We do not agree that Parquet should require one cohort-canonical stored representation for every finite numerical value. Although 1.5, 1.50, and 1.500 are numerically equal, their different quantum can carry information that some applications must preserve. Engines can provide numerical operations alongside IEEE-defined cohort-sensitive operations such as sameQuantum and totalOrder, the latter often exposed as compareTotal. These operations are only possible if the storage format retains the original representation. Canonicalizing on write permanently removes that option. A concrete example is HL7 FHIR healthcare data (https://hl7.org/fhir/datatypes.html). FHIR explicitly specifies that decimal precision is significant: 0.010 is different from 0.01, and implementations must preserve that distinction. This requirement concerns faithful interchange and presentation rather than numerical equality: 0.010 and 0.01 have the same numerical value, but FHIR requires their represented precision to survive a round-trip. A healthcare data lake using Parquet as an interchange or archival format must therefore retain the original quantum to reconstruct standards-compliant FHIR resources. Canonicalizing on write would make that round-trip impossible. Beyond FHIR, this behavior appears in other decimal arithmetic ecosystems. ISO/IEC TS 18661-2 (https://open-std.org/JTC1/SC22/WG14/www/docs/n1912.pdf), which specifies C language support for decimal floating-point arithmetic, explicitly states that 1.0 and 1.00 are distinguishable. The language-independent General Decimal Arithmetic specification (https://speleotrove.com/decimal/daops.html), which underpins several decimal arithmetic libraries, preserves trailing zeros unless the caller explicitly requests `reduce`. The same principle appears independently in measurement and reporting standards. ISO 80000-1 (https://www.iso.org/home.isoDocumentsDownload.do?t=0bc30Dw3Fis0Kmeb_a6O_bzO2QP14DxJAxwtmaRLmv-WwspRuGQ-iHB6VgCGRJnU), NIST SP 811 (https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-7-rules-and-style-conventions-expressing-values), and ASTM E29 (https://www.galvanizeit.com/uploads/resources/ASTM-E-29-yr-13.pdf) all treat trailing zeros after a decimal point as significant in reported values. The Bloom-filter example identifies a real correctness concern, but optional Bloom-filter support should not require writers to discard quantum information from stored values. If Bloom-filter support is deemed necessary, Parquet could canonicalize only the numerical hash key. For example: Stored representation Numerical Bloom-filter key 1.5 = (15, -1) (15, -1) 1.50 = (150, -2) (15, -1) 1.500 = (1500, -3) (15, -1) Writers and readers would hash the same normalized numerical form. This keeps numerical-equality Bloom filters correct without changing the stored representation. Otherwise, readers could disable Bloom-filter pruning for DECFLOAT equality. Special values ------------------- We agree that positive and negative infinity and NaN should be representable. We think the same faithful-round-trip principle should apply to signed zero, signaling NaNs, and NaN payloads. An engine does not need to expose every distinction to its users. It may normalize or reject some values, or provide traps, similar to Firebird’s SET DECFLOAT TRAPS (https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref50-management.html), controlling whether division by zero, overflow, invalid operations, or signaling NaN produce a value or raise an error. Those are execution policies. A concrete example of signaling NaN’s automatic fail-on-use behavior is a bank preparing its daily foreign-exchange rates. Each currency pair must receive a rate from a market-data provider before trades can be valued. The bank could initialize rates that have not yet arrived to signaling NaN and periodically checkpoint the table to Parquet: EUR → USD = 1.17 GBP → USD = 1.35 JPY → USD = sNaN The valuation job enables the InvalidOperation trap. If it accidentally attempts to convert a JPY payment before the rate has arrived, multiplying the payment by sNaN fails immediately at the point of use. Merely copying, checkpointing, or inspecting the table does not trigger the exception. A NaN payload could additionally retain a diagnostic code identifying the missing or invalid data source. Parquet should reproduce the values written to it rather than normalize away supported distinctions. If it converts sNaN to quiet NaN, removes NaN payloads, or collapses -0 into +0, engines that support those values can no longer round-trip them faithfully. Whether an engine exposes or uses these distinctions should remain an engine-level decision; the storage format should preserve them rather than make an irreversible decision for every consumer. Performance ------------------ We take the reported performance differences seriously, but the current results do not isolate the physical representation. They compare Intel libbid with an optimized Snowflake implementation while potentially also varying precision, semantics, algorithms, input distributions, and fast-path behavior. Our own benchmarks reinforce this concern: libbid and Boost.Decimal both use BID, yet differ by several times on some operations. Large performance gaps can therefore arise from implementation choices even when the representation is identical. This does not rule out an inherent advantage in Snowflake’s representation, but the current comparison does not isolate it. Could you share the implementation or enough of the benchmark harness to reproduce the results, including: - input and exponent distributions - precision and rounding configuration - treatment of cohorts and special values - the percentage of operations using the optimized path versus fallback - compiler and optimization settings - results on both ARM and x86? We should compare equivalent value domains and semantics. For Parquet, we should also measure encoding, decoding, comparisons, numerical hashing, statistics, predicate evaluation, compression, and end-to-end scan and aggregation performance, not only arithmetic. We should also separate the Parquet encoding from an engine’s runtime representation. An engine could decode BID once into its preferred coefficient/exponent structure, execute an expression pipeline using that representation, and encode only at the output boundary. A benchmark comparing that approach would help determine whether the reported cost is inherent to the interchange encoding or to using packed BID for every intermediate result. Once those requirements are settled, we can evaluate the candidate representations against the same semantics using reproducible benchmarks. Best, Costas Zarifis
