Hi Costas, Thanks for putting this together. Let's continue the discussion in the doc. Will have a look very soon.
Best, Thomas On Fri, Oct 2, 2026 at 6:51 AM Costas Zarifis <[email protected]> wrote: > Hi all, > > Following Micah’s suggestion, we consolidated the discussion into a formal > proposal: > > Public Proposal: Extensible Decimal Floating-Point Type: > https://docs.google.com/document/d/1cHzlsYH_-JBkcvHLXKSJXMAZ0zwRz3jcnIl4izE1UFQ/edit?usp=sharing > > The proposal takes a clear position in favor of: > - the complete IEEE 754 decimal value model; > - one logical type that preserves the supplied representation; > - content-derived canonicality metadata at the column-chunk level; > - numerical equality and canonical keys for hashing and Bloom filters; and > - IEEE `totalOrder` for representation-sensitive ordering and statistics. > > The physical representation remains to be selected. The document specifies > generalized BID as one candidate, including decimal64, decimal128, and a > 43-digit BID160 profile. It also lists the revisions needed for the > existing significand-and-exponent proposal to become a complete, comparable > alternative. > > Once we agree on the semantic contract, the remaining candidates can be > evaluated under equivalent semantics using public conformance tests, > validation vectors, and benchmarks. > > Please review the proposal and share any correctness, interoperability, or > implementation concerns. We are sharing it now so everyone has time to > review it before our planned discussion. > > Best, > Costas > > On Thu, Sep 24, 2026 at 10:25 AM Micah Kornfield <[email protected]> > wrote: > >> Hi Thomas, >> >> - An empty table has a schema but no existing values from which to derive >>> that policy. >> >> >> The engine can choose the policy; my argument is that the shared schema >>> should communicate it across independent engines and connectors, rather >>> than require separate configuration. A logical-type parameter would express >>> that write-time obligation, while statistics would describe the actual data >>> for read optimization. >> >> >> I think the misalignment we have is that Parquet is not a table format. >> An "empty" table means the absence of a parquet file so no schema would be >> available to coordinate writes. This is partially what I meant by a higher >> level. I could see the argument for documenting the contract at the table >> level, but I don't think it needs to flow down to parquet, as an individual >> file/schema is not something we really care about once the file is written. >> >> As an analogy, there are a lot of other properties we want to specify >> about how writes to a particular column happen (e.g. compression and >> encoding). These are configured separately at the parquet level via an API >> and don't end up in the schema. Table formats have conventions for >> persistently configuring them. >> >> Your point that Parquet cannot prevent these conversions is precisely why >>> we favor excluding these special values from the interoperable type. >>> Documenting an engine's limitation does not alert a downstream consumer >>> that a safety-relevant distinction has already been lost. >> >> >> It isn't Parquet's job to place arbitrary limits on a type for the safety >> of the end-users of a particular engine. Again, I can see higher level >> systems consuming/writing parquet making choices about limiting values, but >> arbitrarily limiting the type does not make sense to me at the file layer. >> >> As an example from traditional floating point numbers. Some systems >> (e.g. postgres) decide to collapse -0.0 and 0.0 to the same bucket on >> aggregation but still persist the values exactly. >> >> Cheers, >> Micah >> >> >> >> >> >> On Thu, Sep 24, 2026 at 8:44 AM Thomas Kissinger < >> [email protected]> wrote: >> >>> Hi Micah, >>> >>> Thank you. >>> *1. Canonicalization: statistics describe data, not writer intent* >>> >>> I think the distinction between read-time optimization and the >>> write-time contract is central here. I agree that statistics can identify >>> safe read optimizations. However, they cannot tell an independent writer >>> whether it should canonicalize incoming values or preserve their quantum: >>> >>> - An empty table has a schema but no existing values from which to >>> derive that policy. >>> - Two columns may both contain only 1.5 and 2.7, yet one requires >>> canonical values and the other requires quantum preservation. Their >>> canonicality statistics are identical. When appending 1.50, the required >>> behavior differs: store 1.5 in the first case and retain 1.50 in the second. >>> >>> A writer must therefore not interpret “all existing values are >>> canonical” as permission to canonicalize new values. Always preserving the >>> supplied representation avoids that mistake, but leaves the canonical-only >>> contract entirely outside Parquet. The engine can choose the policy; my >>> argument is that the shared schema should communicate it across independent >>> engines and connectors, rather than require separate configuration. A >>> logical-type parameter would express that write-time obligation, while >>> statistics would describe the actual data for read optimization. >>> >>> *2. Special values: explicit state is safer for interoperability* >>> >>> We support broad value coverage as a general design principle. However, >>> we do not support including signaling NaN and negative zero in this case. >>> Your examples motivating them encode sensitive application state, not >>> merely numerical values, and that state can disappear silently when passing >>> through engines that do not preserve these distinctions. >>> >>> Consider your financial example of initializing unavailable exchange >>> rates to signaling NaN. If an intermediate engine quietly converts those >>> values to quiet NaN, downstream arithmetic no longer triggers the expected >>> InvalidOperation trap. Likewise, converting -0 to +0 removes information a >>> downstream application may rely on. The rewritten Parquet file can be >>> valid, with accurate statistics, without revealing either loss. >>> >>> Your point that Parquet cannot prevent these conversions is precisely >>> why we favor excluding these special values from the interoperable type. >>> Documenting an engine's limitation does not alert a downstream consumer >>> that a safety-relevant distinction has already been lost. >>> >>> We would instead represent such state explicitly in auxiliary >>> columns—for example, a nullable exchange rate alongside a status such as >>> VALID, MISSING, or INVALID, with an optional diagnostic code. An >>> application relying on the sign of zero could likewise store that >>> distinction explicitly. These fields still require application-level >>> validation, but use ordinary, widely supported types and make the state >>> directly inspectable, rather than depend on special-value behavior that may >>> change across engines. >>> >>> Best, >>> Thomas >>> >>> quick note: won't be able to reply before next Thu. >>> >>> On Wed, Sep 23, 2026 at 6:08 PM Micah Kornfield <[email protected]> >>> wrote: >>> >>>> Hi Thomas, >>>> >>>>> We expect quantum-preserving round-trips to serve a minority of use >>>>> cases, while most analytical workloads need only numerical values. >>>>> Requiring cohort handling in these common cases risks substantial overhead >>>>> in equality and hashing without adding value. Statistics can identify fast >>>>> paths, but describe what was written rather than prescribe whether a >>>>> column >>>>> must be canonicalized. A logical-type parameter would explicitly define >>>>> the >>>>> writer's obligation to canonicalize or preserve quantum; statistics alone >>>>> do not establish that contract. >>>> >>>> >>>> I agree this can be useful but it seems like a decision taken at a >>>> higher level? From a parquet persistence perspective I think this breaks >>>> down as follows: >>>> 1. An engine/connector decides to hand values to parquet. If we are >>>> in the regime that canonicalization is required the engine/connector is >>>> making that decision and only providing canonical values. This will show >>>> up in statistics. For optimizing stats collection it seems like we could >>>> expose APIs in parquet to assume values are canonicalized if the overhead >>>> of checking is deemed too expensive. But that seems like an API level >>>> concern and not a schema level concern. >>>> 2. When reading, an engine/connector can look at the statistics to >>>> determine if everything is canocilized. Having the type parameter seems >>>> superfluous? >>>> >>>> Given this model could you expand on why you think the parameterization >>>> has to happen at the parquet schema level? >>>> >>>> Our concern is silent information loss when data passes through a >>>>> non-supporting engine. Such an engine may quiet a signaling NaN or turn >>>>> negative zero into ordinary zero when reading and rewriting the data. A >>>>> downstream supporting reader could then rely on the result as a faithful >>>>> representation of the original values, without knowing that these >>>>> meaningful distinctions were discarded. Metadata describing the rewritten >>>>> values would not reveal that loss. >>>> >>>> >>>> I think this is outside the scope of Parquet. We've always taken the >>>> stance that Parquet should be able to support taking in a full range of >>>> values, and be able to return them losslessly. If an engine that doesn't >>>> support the full range chooses to coerce the values, that is its choice and >>>> it should document the limitation. >>>> >>>> Best, >>>> Micah >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Sep 21, 2026 at 8:00 AM Thomas Kissinger < >>>> [email protected]> wrote: >>>> >>>>> Hi Costas, >>>>> >>>>> Sorry for the late reply. I was on vacation last week. >>>>> >>>>> Thank you for the thoughtful reply. The statistics proposal sounds >>>>> interesting, and I would like to focus on two remaining points. >>>>> >>>>> *1. Canonical representation* >>>>> We expect quantum-preserving round-trips to serve a minority of use >>>>> cases, while most analytical workloads need only numerical values. >>>>> Requiring cohort handling in these common cases risks substantial overhead >>>>> in equality and hashing without adding value. Statistics can identify fast >>>>> paths, but describe what was written rather than prescribe whether a >>>>> column >>>>> must be canonicalized. A logical-type parameter would explicitly define >>>>> the >>>>> writer's obligation to canonicalize or preserve quantum; statistics alone >>>>> do not establish that contract. >>>>> >>>>> Another option would be to always store a canonical numerical >>>>> component alongside compact quantum information, such as the number of >>>>> trailing zeros removed during canonicalization. Equality and hashing would >>>>> use the canonical component, while the additional information would allow >>>>> the original representation to round-trip. This could reconcile efficient >>>>> numerical operations with quantum preservation at a small storage >>>>> overhead. >>>>> Would you consider this approach? >>>>> >>>>> *2. Signaling NaNs and negative zero* >>>>> Our concern is silent information loss when data passes through a >>>>> non-supporting engine. Such an engine may quiet a signaling NaN or turn >>>>> negative zero into ordinary zero when reading and rewriting the data. A >>>>> downstream supporting reader could then rely on the result as a faithful >>>>> representation of the original values, without knowing that these >>>>> meaningful distinctions were discarded. Metadata describing the rewritten >>>>> values would not reveal that loss. >>>>> >>>>> How would the proposed round-trip contract prevent this loss from >>>>> going unnoticed and misleading downstream readers? >>>>> >>>>> >>>>> Micah, thank you for helping move this forward; since the requirements >>>>> seem to be converging, we would prefer to resolve these remaining points >>>>> here first, then work together on a joint proposal, if that works for >>>>> everyone. >>>>> >>>>> Best, >>>>> Thomas >>>>> >>>>> On Tue, Sep 15, 2026 at 7:18 PM Micah Kornfield <[email protected]> >>>>> wrote: >>>>> >>>>>> It would be good to move this conversation along. I think it is >>>>>> reasonable to potentially make the normalization part of statistics. >>>>>> >>>>>> To move this conversation forward I would suggest the following: >>>>>> 1. Given the amount of back and forth, I think we should translate >>>>>> the discussion so far into a formal proposal ( >>>>>> https://github.com/apache/parquet-format/blob/master/proposals/README.md) >>>>>> with a proper design doc. Costas, could you do this? >>>>>> 2. Identify open issues and how to close on them. I think reading >>>>>> through the thread the two main ones that pop out are: >>>>>> a. Normalization and how to represent it (hopefully we can close >>>>>> on this by using statistics). >>>>>> b. Representation of the value in Parquet. In order to close on >>>>>> I think we need to solidify any alternatives ieee754 and how they would >>>>>> work. Then we can run appropriate benchmarks to help make a decision. >>>>>> Thomas if you still want to consider an alternative please write up a >>>>>> design that we can include in the proposal. >>>>>> >>>>>> Best, >>>>>> Micah >>>>>> >>>>>> On Fri, Sep 4, 2026 at 8:34 PM Costas Zarifis via dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Thomas, >>>>>>> >>>>>>> Thank you for the detailed response. I think we are close to >>>>>>> agreement on >>>>>>> the requirements. The main remaining question is whether canonicality >>>>>>> should be represented as a property of the logical type or as a >>>>>>> property of >>>>>>> the encoded data. >>>>>>> >>>>>>> We agree on the following: >>>>>>> - Precision should be parameterized rather than fixed at 38, allowing >>>>>>> narrower encodings and a defined path to wider precisions. >>>>>>> - The type should cover the complete finite decimal128 range. >>>>>>> - At minimum, positive and negative infinity and a canonical NaN >>>>>>> should be >>>>>>> supported. >>>>>>> - Logical equality, hashing, Bloom filters, ordering, and statistics >>>>>>> must >>>>>>> remain consistent regardless of the stored representation. >>>>>>> - The physical representation should be selected only after comparing >>>>>>> candidates under equivalent semantics through public benchmarks and >>>>>>> validation vectors. >>>>>>> >>>>>>> We do not think normalization should parameterize the logical type >>>>>>> solely >>>>>>> to expose an optimization opportunity. Members of the same finite >>>>>>> cohort, >>>>>>> such as 1.5 and 1.50, have the same numerical identity and must >>>>>>> behave >>>>>>> consistently for equality, hashing, grouping, ordering, and Bloom >>>>>>> filters. >>>>>>> At the same time, a Parquet writer that is supplied with a particular >>>>>>> coefficient and exponent should preserve that representation. >>>>>>> >>>>>>> These requirements do not conflict. Implementations can derive a >>>>>>> cohort-invariant numerical key for equality and hashing without >>>>>>> rewriting >>>>>>> the stored value. Canonicalizing that derived key is always safe; >>>>>>> replacing >>>>>>> the stored value 1.50 with 1.5 is not safe when quantum must >>>>>>> round-trip. >>>>>>> >>>>>>> An engine may expose only numerical semantics and therefore supply >>>>>>> already-canonical values to the writer, but that should not create a >>>>>>> second >>>>>>> Parquet logical type. Once a DECFLOAT value with a particular >>>>>>> coefficient >>>>>>> and exponent is supplied, a conforming writer must preserve it. >>>>>>> >>>>>>> At the same time, we agree with your concern that preserving cohorts >>>>>>> can >>>>>>> add work to equality and hash-based operations. That work can be >>>>>>> avoided >>>>>>> when the encoded data is known to satisfy stronger invariants, >>>>>>> without >>>>>>> making those invariants part of the logical type. >>>>>>> >>>>>>> We therefore suggest defining content-derived properties that >>>>>>> writers can >>>>>>> record at the file or row-group level (or another agreed physical >>>>>>> scope). >>>>>>> These could indicate, for example, that: >>>>>>> >>>>>>> 1. every finite value uses the defined cohort-canonical >>>>>>> representation >>>>>>> 2. no negative zero occurs; >>>>>>> 3. no signaling NaN occurs; or >>>>>>> 4. only the canonical NaN occurs. >>>>>>> >>>>>>> Implementations could inspect these properties before decoding the >>>>>>> column >>>>>>> data and select an optimized path when safe. For example, if every >>>>>>> finite >>>>>>> value has a unique canonical encoding and the relevant special cases >>>>>>> are >>>>>>> absent, equality and hashing may avoid per-value canonicalization. >>>>>>> If the >>>>>>> properties are false, absent or unknown, implementations must use the >>>>>>> general numerical path performing canonicalization. The properties >>>>>>> only >>>>>>> inform optimization; they do not change the values’ logical behavior. >>>>>>> >>>>>>> There remains an important distinction between observing that a >>>>>>> particular >>>>>>> file or row group contains only canonical values and enforcing that a >>>>>>> column may never contain anything else. A mandatory canonical mode >>>>>>> would >>>>>>> prevent representational round-tripping even when the source engine >>>>>>> retained the necessary information. A content-derived property >>>>>>> provides the >>>>>>> optimized path whenever the data already satisfies the invariant >>>>>>> without >>>>>>> changing the logical type or authorizing information loss. >>>>>>> >>>>>>> This requires defining new metadata or statistics, including their >>>>>>> precise >>>>>>> meaning, scope, and writer requirements. Concretely, a writer could >>>>>>> compute >>>>>>> the property as follows: >>>>>>> >>>>>>> allValuesCanonical = true >>>>>>> For each value: >>>>>>> if value is not canonical: >>>>>>> allValuesCanonical = false >>>>>>> >>>>>>> This flag at the end, stores one of three states: >>>>>>> >>>>>>> 1. true: every value is normalized >>>>>>> 2. false: at least one value is not normalized >>>>>>> 3. absent/unknown: the writer did not establish the property >>>>>>> >>>>>>> For example: >>>>>>> >>>>>>> File A: 1.5, 2.7, 3.9 >>>>>>> allValuesCanonical = true >>>>>>> File B: 1.5, 1.50, 3.9 >>>>>>> allValuesCanonical = false >>>>>>> >>>>>>> This could be stored in one of these places: >>>>>>> >>>>>>> 1. Table/column: one property for the entire table >>>>>>> 2. File: each Parquet file reports its own property >>>>>>> 3. Row group: different parts of a file can report different >>>>>>> properties >>>>>>> 4. Page: even finer-grained, but with greater metadata overhead >>>>>>> >>>>>>> We suggest capturing the remaining requirements in a small decision >>>>>>> matrix: >>>>>>> >>>>>>> 1. Supported precision values and range. >>>>>>> 2. The representation and round-trip contract for finite cohorts and >>>>>>> special values. >>>>>>> 3. Numerical equality and normative canonical-key semantics. >>>>>>> 4. Content-derived canonicality properties, including their scope and >>>>>>> missing/unknown behavior. >>>>>>> 5. Whether canonicality is an observed property or an enforced >>>>>>> invariant. >>>>>>> 6. Statistics and numerical ordering. >>>>>>> 7. Invalid, reserved, and non-canonical encoding handling. >>>>>>> 8. Required benchmarks, validation vectors, and physical-format >>>>>>> artifacts. >>>>>>> >>>>>>> Once that matrix converges, we would be happy to work jointly on the >>>>>>> benchmark and validation suite. At that point, the candidate physical >>>>>>> representations can be evaluated against the same requirements >>>>>>> rather than >>>>>>> embedding different semantics in each comparison. >>>>>>> >>>>>>> Best, >>>>>>> Costas >>>>>>> >>>>>>> On Tue, Aug 25, 2026 at 6:50 AM Thomas Kissinger via dev < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>> > Hi Costas, >>>>>>> > >>>>>>> > Thank you for the thoughtful reply and for separating the >>>>>>> requirements from >>>>>>> > the physical representation. Starting with the requirements is the >>>>>>> right >>>>>>> > approach, and we are confident that we can find a solution >>>>>>> covering the >>>>>>> > important use cases on both sides. >>>>>>> > >>>>>>> > Thank you also for sharing the Spark SPIP. Seeing the proposed >>>>>>> engine >>>>>>> > semantics alongside the Parquet representation is very helpful. >>>>>>> Although >>>>>>> > Spark and Parquet have separate decision processes, both projects >>>>>>> would >>>>>>> > benefit if these efforts converge on shared requirements rather >>>>>>> than >>>>>>> > develop alongside each other with different assumptions. In >>>>>>> particular, >>>>>>> > aligning on precision and range, which numerical and >>>>>>> representational >>>>>>> > distinctions must survive a round-trip, and which special values >>>>>>> are >>>>>>> > supported before either design is finalized would avoid unnecessary >>>>>>> > conversion and interoperability gaps. >>>>>>> > >>>>>>> > Our current view is the following. >>>>>>> > Precision and range >>>>>>> > >>>>>>> > We agree that the type should not impose a permanent 38-digit >>>>>>> ceiling and >>>>>>> > should cover at least the complete finite decimal128 range. >>>>>>> Narrower and >>>>>>> > wider precisions should be possible, with the exact supported set >>>>>>> decided >>>>>>> > when evaluating the physical representations. >>>>>>> > >>>>>>> > This requirement also exposes a mismatch with the generalized IEEE >>>>>>> > interchange layout. Its precision ladder moves from 34 to 43 >>>>>>> digits around >>>>>>> > the established 38-digit SQL boundary, and from 70 to 79 around the >>>>>>> > 76-digit boundary. Current Intel and Boost implementations stop at >>>>>>> > decimal128, and we are not aware of an interoperable production >>>>>>> decimal160 >>>>>>> > codec. Signed 128- and 256-bit integer significands support >>>>>>> precisions 38 >>>>>>> > and 76 directly, aligning more naturally with existing database >>>>>>> decimal >>>>>>> > representations. >>>>>>> > Quantum, equality, and hashing >>>>>>> > >>>>>>> > We agree that some applications need to preserve the original >>>>>>> quantum, and >>>>>>> > FHIR is a useful example. The question is not whether this >>>>>>> capability >>>>>>> > should exist, but whether it should be mandatory for every column. >>>>>>> For >>>>>>> > workloads that require only ordinary numerical semantics, >>>>>>> preserving >>>>>>> > cohorts adds work to equality and hash-based operations without >>>>>>> providing >>>>>>> > additional value. >>>>>>> > >>>>>>> > When cohort members are retained, physical identity and numerical >>>>>>> identity >>>>>>> > diverge. Equality checks must handle multiple encodings of the >>>>>>> same value, >>>>>>> > while hash joins, grouping, DISTINCT, and partitioning must derive >>>>>>> a >>>>>>> > cohort- invariant numerical key for every processed value or >>>>>>> maintain an >>>>>>> > additional canonical key. That adds work and state to fundamental >>>>>>> database >>>>>>> > paths. Because Parquet Bloom filters currently hash the PLAIN >>>>>>> encoding, a >>>>>>> > quantum-preserving mode would require a normative cohort-invariant >>>>>>> key >>>>>>> > shared by writers and readers; any disagreement could cause >>>>>>> incorrect >>>>>>> > pruning. >>>>>>> > >>>>>>> > Preserving cohort members also increases physical cardinality: >>>>>>> numerically >>>>>>> > equal values may be stored as different byte sequences. This can >>>>>>> create >>>>>>> > additional dictionary entries and reduce repeated byte patterns >>>>>>> available >>>>>>> > to compression. The actual size impact will depend on the data >>>>>>> distribution >>>>>>> > and should be included in the benchmark. >>>>>>> > >>>>>>> > We therefore propose making quantum preservation an explicit >>>>>>> parameter of >>>>>>> > the logical type, with canonical representation as the default. A >>>>>>> canonical >>>>>>> > mode would require one representation per finite numerical value, >>>>>>> while a >>>>>>> > quantum- preserving mode would retain the supplied exponent for >>>>>>> > applications requiring faithful round-tripping. Both modes would >>>>>>> use >>>>>>> > ordinary numerical equality; the parameter would define the >>>>>>> representation >>>>>>> > round-trip contract. This makes the semantic and performance >>>>>>> tradeoff >>>>>>> > explicit rather than imposing either choice on every consumer. >>>>>>> > Special values >>>>>>> > >>>>>>> > We agree on positive and negative infinity and a canonical NaN. We >>>>>>> are less >>>>>>> > convinced that signed zero, signaling NaNs, and NaN payloads >>>>>>> belong in an >>>>>>> > interoperable Parquet type, since many important target type >>>>>>> systems cannot >>>>>>> > represent them reliably. >>>>>>> > >>>>>>> > Signaling NaN is difficult to reconcile with an interoperable >>>>>>> columnar >>>>>>> > format because its behavior depends on how a value is inspected. >>>>>>> IEEE >>>>>>> > numerical comparisons signal invalid operation for an sNaN, while >>>>>>> > totalOrder, which is appropriate for statistics, does not. Parquet >>>>>>> uses >>>>>>> > values through several paths—statistics and pruning, dictionary >>>>>>> processing, >>>>>>> > and row-level predicate evaluation—that may therefore treat the >>>>>>> same sNaN >>>>>>> > differently. With traps enabled, observable query behavior could >>>>>>> even >>>>>>> > depend on whether an optimization path is used. >>>>>>> > >>>>>>> > Supporting sNaN would consequently require precise rules for >>>>>>> statistics, >>>>>>> > pruning, predicates, dictionaries, quieting, and round-tripping. >>>>>>> That is >>>>>>> > substantial complexity for an execution mechanism without a >>>>>>> demonstrated >>>>>>> > Parquet interoperability requirement. In the exchange-rate >>>>>>> example, NULL or >>>>>>> > an explicit state column would express missing or invalid input >>>>>>> more >>>>>>> > predictably across engines. >>>>>>> > >>>>>>> > Signed zero creates a similar interoperability problem to >>>>>>> signaling NaN. It >>>>>>> > introduces two encodings for the same numerical zero and therefore >>>>>>> requires >>>>>>> > special treatment in equality, hashing, Bloom filters, ordering, >>>>>>> and >>>>>>> > statistics. Many decimal type systems and conversion paths do not >>>>>>> preserve >>>>>>> > the distinction and silently turn negative zero into ordinary >>>>>>> zero. A >>>>>>> > downstream consumer relying on the sign may then observe different >>>>>>> behavior >>>>>>> > without any indication that information was lost. This is >>>>>>> misleading in an >>>>>>> > interoperable format, so unless a compelling use case is >>>>>>> identified, we >>>>>>> > prefer one canonical zero. >>>>>>> > >>>>>>> > We do not think NaN payloads should be included. IEEE defines their >>>>>>> > representation but not interoperable payload semantics, and does >>>>>>> not fully >>>>>>> > specify payload selection when multiple NaNs participate in an >>>>>>> operation. >>>>>>> > Parquet could preserve the bytes, but that would impose a >>>>>>> round-trip >>>>>>> > obligation for opaque metadata that receiving engines cannot >>>>>>> interpret >>>>>>> > consistently. Diagnostic information is more portable and >>>>>>> queryable in a >>>>>>> > separate column. >>>>>>> > Performance and benchmarks >>>>>>> > >>>>>>> > We agree that, once the requirements are finalized, we should work >>>>>>> together >>>>>>> > on a public benchmark comparing equivalent semantics and >>>>>>> documenting >>>>>>> > distributions, rounding behavior, fast paths, fallbacks, compiler >>>>>>> settings, >>>>>>> > and hardware. It should cover fundamental database operations, >>>>>>> including >>>>>>> > encoding, decoding, cohort reduction and canonical-key derivation, >>>>>>> > equality, and arithmetic. >>>>>>> > >>>>>>> > Until the existing measurements can be reproduced under equivalent >>>>>>> > semantics, they should be treated as directional evidence rather >>>>>>> than a >>>>>>> > representation-level result. Please feel free to propose an initial >>>>>>> > benchmark design already; it could provide a useful foundation for >>>>>>> the >>>>>>> > joint version. >>>>>>> > Storage and runtime representation >>>>>>> > >>>>>>> > We agree that engines can decode the Parquet representation once >>>>>>> and use a >>>>>>> > different runtime representation. This amortizes conversion across >>>>>>> long >>>>>>> > pipelines but does not eliminate representation-dependent work. >>>>>>> Values >>>>>>> > surviving pruning must still be interpreted or decoded for >>>>>>> row-level >>>>>>> > filters, numerical hashing, statistics construction on write, or >>>>>>> runtime >>>>>>> > materialization. The physical representation therefore remains >>>>>>> relevant to >>>>>>> > scan-heavy workloads. >>>>>>> > Validation >>>>>>> > >>>>>>> > We likewise agree that, once the requirements are finalized, we >>>>>>> should >>>>>>> > jointly develop a public validation suite independently of the >>>>>>> performance >>>>>>> > benchmark. Please feel free to propose an initial set of test >>>>>>> vectors >>>>>>> > already as a foundation. It should eventually cover canonical and >>>>>>> > quantum-preserving values, precision boundaries, special values, >>>>>>> > non-canonical, reserved, and invalid encodings as applicable, >>>>>>> equality, >>>>>>> > hashing, ordering and statistics, and cross-implementation >>>>>>> round-trips. >>>>>>> > >>>>>>> > Thanks again for moving the discussion toward explicit >>>>>>> requirements. We >>>>>>> > already agree on many important points: support for at least 38 >>>>>>> digits with >>>>>>> > a path to wider precisions, the decimal128 finite range, >>>>>>> infinities and a >>>>>>> > canonical NaN, and public benchmarks and test vectors. The >>>>>>> remaining >>>>>>> > discussion is focused on whether quantum preservation should be >>>>>>> optional, >>>>>>> > which additional IEEE distinctions belong in an interoperable >>>>>>> database >>>>>>> > type, and which physical representation best satisfies the agreed >>>>>>> > requirements. We believe an extensible type with an explicit >>>>>>> > quantum-preservation choice offers a practical path forward. >>>>>>> > >>>>>>> > Best, Thomas >>>>>>> > >>>>>>> > >>>>>>> > On Fri, Aug 14, 2026 at 10:38 PM Costas Zarifis via dev < >>>>>>> > [email protected]> wrote: >>>>>>> > >>>>>>> > > Hi Thomas, >>>>>>> > > >>>>>>> > > Thank you for laying out the requirements explicitly. We agree >>>>>>> that >>>>>>> > > requirements should come before selecting a physical >>>>>>> representation. >>>>>>> > > There is substantial agreement, but we see several requirements >>>>>>> > > differently. >>>>>>> > > >>>>>>> > > For context on engine integration, we have also published a draft >>>>>>> > > Spark Project Improvement Proposal (SPIP): Add the DECFLOAT data >>>>>>> type >>>>>>> > > ( >>>>>>> > > >>>>>>> > >>>>>>> https://docs.google.com/document/d/1mTSOoCvX7yYMSa6CpI6OPS-MQZocPxUtrrFOEiZjCPk/edit >>>>>>> > > ). >>>>>>> > > The SPIP covers the proposed Spark SQL type and execution >>>>>>> semantics, >>>>>>> > > while this discussion focuses on the portable Parquet >>>>>>> representation. >>>>>>> > > >>>>>>> > > Before choosing BID or a new 18-byte format, we suggest first >>>>>>> agreeing >>>>>>> > > on the following: >>>>>>> > > >>>>>>> > > - Precision: We agree that the format must losslessly support >>>>>>> 38-digit >>>>>>> > > values. Your proposal targets exactly 38 digits, while we think >>>>>>> the >>>>>>> > > format should provide a defined path to both narrower and wider >>>>>>> > > precisions. >>>>>>> > > - Range: We agree that the format should cover at least the >>>>>>> complete >>>>>>> > > finite decimal128 range. >>>>>>> > > - Cohorts: Your proposal requires one canonical representation >>>>>>> per >>>>>>> > > numerical value. We think the original coefficient, exponent, and >>>>>>> > > quantum should round-trip. >>>>>>> > > - Special values: We agree on infinity and NaN. We additionally >>>>>>> think >>>>>>> > > engines supporting signed zero, signaling NaNs, and NaN payloads >>>>>>> > > should be able to round-trip them. >>>>>>> > > - Hashing and equality: We agree that numerically equal values >>>>>>> must >>>>>>> > > behave consistently in equality and hash-based operations. We do >>>>>>> not >>>>>>> > > think that requires canonicalizing the stored representation. >>>>>>> > > - Performance: We take the reported results seriously, but think >>>>>>> the >>>>>>> > > candidate representations should be compared using equivalent >>>>>>> > > semantics, public benchmarks, and independent implementations. >>>>>>> > > - Validation: Correctness and interoperability should be >>>>>>> validated >>>>>>> > > using public test vectors, separately from performance >>>>>>> evaluation. >>>>>>> > > >>>>>>> > > The sections below provide the rationale for each position. >>>>>>> > > >>>>>>> > > Precision and range >>>>>>> > > --------------------------- >>>>>>> > > >>>>>>> > > We agree that a Parquet decimal-floating-point type must >>>>>>> losslessly >>>>>>> > > represent every numerical value in the DECIMAL(38,s) domain. >>>>>>> > > >>>>>>> > > We do not think the format should necessarily stop at exactly 38 >>>>>>> > > digits, however. Precision 38 is an important interoperability >>>>>>> > > boundary established by an original crop of DBMS, which does not >>>>>>> > > establish that present or future requirements stop at 38 digits. >>>>>>> > > >>>>>>> > > Many open source or major newer DBMS have higher limits: >>>>>>> > > >>>>>>> > > - PostgreSQL >>>>>>> > > Type: NUMERIC unconstrained >>>>>>> > > Max precision: 131,072 before decimal / 16,383 after; declared >>>>>>> > > NUMERIC(p,s) up to p=1000 >>>>>>> > > Source: >>>>>>> https://www.postgresql.org/docs/current/datatype-numeric.html >>>>>>> > > >>>>>>> > > - MySQL / MariaDB / SingleStore >>>>>>> > > Type: DECIMAL >>>>>>> > > Max precision: 65 >>>>>>> > > Source: >>>>>>> https://dev.mysql.com/doc/refman/8.0/en/fixed-point-types.html >>>>>>> > > >>>>>>> > > - BigQuery >>>>>>> > > Type: BIGNUMERIC >>>>>>> > > Max precision: ~76.8 (scale 38) >>>>>>> > > Source: >>>>>>> > > >>>>>>> https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types >>>>>>> > > >>>>>>> > > - ClickHouse >>>>>>> > > Type: Decimal256 >>>>>>> > > Max precision: 76 >>>>>>> > > Source: >>>>>>> https://clickhouse.com/docs/sql-reference/data-types/decimal >>>>>>> > > >>>>>>> > > - Oracle >>>>>>> > > Type: unconstrained NUMBER >>>>>>> > > Max precision: ~40 (parameterized 38) >>>>>>> > > Source: >>>>>>> > > >>>>>>> > >>>>>>> https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlqr/Data-Types.html >>>>>>> > > >>>>>>> > > - Trino ≥480 >>>>>>> > > Type: NUMBER >>>>>>> > > Max precision: >= 50 >>>>>>> > > Source: >>>>>>> https://trino.io/docs/current/language/types.html#number >>>>>>> > > >>>>>>> > > While we can opine on the value of precision past a certain >>>>>>> limit, we >>>>>>> > > cannot pass judgement on whether the cut-off is 34, 38, 43 or >>>>>>> > > something higher. >>>>>>> > > >>>>>>> > > Therefore Parquet should avoid making any such precision a >>>>>>> permanent >>>>>>> > > semantic ceiling. >>>>>>> > > >>>>>>> > > Faithful round-tripping and cohorts >>>>>>> > > ---------------------------------------------- >>>>>>> > > >>>>>>> > > We do not agree that Parquet should require one cohort-canonical >>>>>>> > > stored representation for every finite numerical value. >>>>>>> > > >>>>>>> > > Although 1.5, 1.50, and 1.500 are numerically equal, their >>>>>>> different >>>>>>> > > quantum can carry information that some applications must >>>>>>> preserve. >>>>>>> > > Engines can provide numerical operations alongside IEEE-defined >>>>>>> > > cohort-sensitive operations such as sameQuantum and totalOrder, >>>>>>> the >>>>>>> > > latter often exposed as compareTotal. These operations are only >>>>>>> > > possible if the storage format retains the original >>>>>>> representation. >>>>>>> > > Canonicalizing on write permanently removes that option. >>>>>>> > > >>>>>>> > > A concrete example is HL7 FHIR healthcare data >>>>>>> > > (https://hl7.org/fhir/datatypes.html). FHIR explicitly >>>>>>> specifies that >>>>>>> > > decimal precision is significant: 0.010 is different from 0.01, >>>>>>> and >>>>>>> > > implementations must preserve that distinction. This requirement >>>>>>> > > concerns faithful interchange and presentation rather than >>>>>>> numerical >>>>>>> > > equality: 0.010 and 0.01 have the same numerical value, but FHIR >>>>>>> > > requires their represented precision to survive a round-trip. A >>>>>>> > > healthcare data lake using Parquet as an interchange or archival >>>>>>> > > format must therefore retain the original quantum to reconstruct >>>>>>> > > standards-compliant FHIR resources. Canonicalizing on write >>>>>>> would make >>>>>>> > > that round-trip impossible. >>>>>>> > > >>>>>>> > > Beyond FHIR, this behavior appears in other decimal arithmetic >>>>>>> > > ecosystems. ISO/IEC TS 18661-2 >>>>>>> > > (https://open-std.org/JTC1/SC22/WG14/www/docs/n1912.pdf), which >>>>>>> > > specifies C language support for decimal floating-point >>>>>>> arithmetic, >>>>>>> > > explicitly states that 1.0 and 1.00 are distinguishable. The >>>>>>> > > language-independent General Decimal Arithmetic specification >>>>>>> > > (https://speleotrove.com/decimal/daops.html), which underpins >>>>>>> several >>>>>>> > > decimal arithmetic libraries, preserves trailing zeros unless the >>>>>>> > > caller explicitly requests `reduce`. The same principle appears >>>>>>> > > independently in measurement and reporting standards. ISO 80000-1 >>>>>>> > > ( >>>>>>> > > >>>>>>> > >>>>>>> https://www.iso.org/home.isoDocumentsDownload.do?t=0bc30Dw3Fis0Kmeb_a6O_bzO2QP14DxJAxwtmaRLmv-WwspRuGQ-iHB6VgCGRJnU >>>>>>> > > ), >>>>>>> > > NIST SP 811 ( >>>>>>> > > >>>>>>> > >>>>>>> https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-7-rules-and-style-conventions-expressing-values >>>>>>> > > ), >>>>>>> > > and ASTM E29 ( >>>>>>> > > >>>>>>> https://www.galvanizeit.com/uploads/resources/ASTM-E-29-yr-13.pdf) >>>>>>> > > all treat trailing zeros after a decimal point as significant in >>>>>>> > > reported values. >>>>>>> > > >>>>>>> > > The Bloom-filter example identifies a real correctness concern, >>>>>>> but >>>>>>> > > optional Bloom-filter support should not require writers to >>>>>>> discard >>>>>>> > > quantum information from stored values. If Bloom-filter support >>>>>>> is >>>>>>> > > deemed necessary, Parquet could canonicalize only the numerical >>>>>>> hash >>>>>>> > > key. >>>>>>> > > >>>>>>> > > For example: >>>>>>> > > >>>>>>> > > Stored representation Numerical Bloom-filter key >>>>>>> > > 1.5 = (15, -1) (15, -1) >>>>>>> > > 1.50 = (150, -2) (15, -1) >>>>>>> > > 1.500 = (1500, -3) (15, -1) >>>>>>> > > >>>>>>> > > Writers and readers would hash the same normalized numerical >>>>>>> form. >>>>>>> > > This keeps numerical-equality Bloom filters correct without >>>>>>> changing >>>>>>> > > the stored representation. Otherwise, readers could disable >>>>>>> > > Bloom-filter pruning for DECFLOAT equality. >>>>>>> > > >>>>>>> > > Special values >>>>>>> > > ------------------- >>>>>>> > > >>>>>>> > > We agree that positive and negative infinity and NaN should be >>>>>>> > > representable. We think the same faithful-round-trip principle >>>>>>> should >>>>>>> > > apply to signed zero, signaling NaNs, and NaN payloads. >>>>>>> > > >>>>>>> > > An engine does not need to expose every distinction to its >>>>>>> users. It >>>>>>> > > may normalize or reject some values, or provide traps, similar to >>>>>>> > > Firebird’s SET DECFLOAT TRAPS >>>>>>> > > ( >>>>>>> > > >>>>>>> > >>>>>>> https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref50-management.html >>>>>>> > > ), >>>>>>> > > controlling whether division by zero, overflow, invalid >>>>>>> operations, or >>>>>>> > > signaling NaN produce a value or raise an error. Those are >>>>>>> execution >>>>>>> > > policies. >>>>>>> > > >>>>>>> > > A concrete example of signaling NaN’s automatic fail-on-use >>>>>>> behavior >>>>>>> > > is a bank preparing its daily foreign-exchange rates. Each >>>>>>> currency >>>>>>> > > pair must receive a rate from a market-data provider before >>>>>>> trades can >>>>>>> > > be valued. The bank could initialize rates that have not yet >>>>>>> arrived >>>>>>> > > to signaling NaN and periodically checkpoint the table to >>>>>>> Parquet: >>>>>>> > > >>>>>>> > > EUR → USD = 1.17 >>>>>>> > > GBP → USD = 1.35 >>>>>>> > > JPY → USD = sNaN >>>>>>> > > >>>>>>> > > The valuation job enables the InvalidOperation trap. If it >>>>>>> > > accidentally attempts to convert a JPY payment before the rate >>>>>>> has >>>>>>> > > arrived, multiplying the payment by sNaN fails immediately at the >>>>>>> > > point of use. Merely copying, checkpointing, or inspecting the >>>>>>> table >>>>>>> > > does not trigger the exception. A NaN payload could additionally >>>>>>> > > retain a diagnostic code identifying the missing or invalid data >>>>>>> > > source. >>>>>>> > > >>>>>>> > > Parquet should reproduce the values written to it rather than >>>>>>> > > normalize away supported distinctions. If it converts sNaN to >>>>>>> quiet >>>>>>> > > NaN, removes NaN payloads, or collapses -0 into +0, engines that >>>>>>> > > support those values can no longer round-trip them faithfully. >>>>>>> Whether >>>>>>> > > an engine exposes or uses these distinctions should remain an >>>>>>> > > engine-level decision; the storage format should preserve them >>>>>>> rather >>>>>>> > > than make an irreversible decision for every consumer. >>>>>>> > > >>>>>>> > > Performance >>>>>>> > > ------------------ >>>>>>> > > >>>>>>> > > We take the reported performance differences seriously, but the >>>>>>> > > current results do not isolate the physical representation. They >>>>>>> > > compare Intel libbid with an optimized Snowflake implementation >>>>>>> while >>>>>>> > > potentially also varying precision, semantics, algorithms, input >>>>>>> > > distributions, and fast-path behavior. >>>>>>> > > >>>>>>> > > Our own benchmarks reinforce this concern: libbid and >>>>>>> Boost.Decimal >>>>>>> > > both use BID, yet differ by several times on some operations. >>>>>>> Large >>>>>>> > > performance gaps can therefore arise from implementation choices >>>>>>> even >>>>>>> > > when the representation is identical. This does not rule out an >>>>>>> > > inherent advantage in Snowflake’s representation, but the current >>>>>>> > > comparison does not isolate it. >>>>>>> > > >>>>>>> > > Could you share the implementation or enough of the benchmark >>>>>>> harness >>>>>>> > > to reproduce the results, including: >>>>>>> > > >>>>>>> > > - input and exponent distributions >>>>>>> > > - precision and rounding configuration >>>>>>> > > - treatment of cohorts and special values >>>>>>> > > - the percentage of operations using the optimized path versus >>>>>>> fallback >>>>>>> > > - compiler and optimization settings >>>>>>> > > - results on both ARM and x86? >>>>>>> > > >>>>>>> > > We should compare equivalent value domains and semantics. For >>>>>>> Parquet, >>>>>>> > > we should also measure encoding, decoding, comparisons, numerical >>>>>>> > > hashing, statistics, predicate evaluation, compression, and >>>>>>> end-to-end >>>>>>> > > scan and aggregation performance, not only arithmetic. >>>>>>> > > >>>>>>> > > We should also separate the Parquet encoding from an engine’s >>>>>>> runtime >>>>>>> > > representation. An engine could decode BID once into its >>>>>>> preferred >>>>>>> > > coefficient/exponent structure, execute an expression pipeline >>>>>>> using >>>>>>> > > that representation, and encode only at the output boundary. A >>>>>>> > > benchmark comparing that approach would help determine whether >>>>>>> the >>>>>>> > > reported cost is inherent to the interchange encoding or to using >>>>>>> > > packed BID for every intermediate result. >>>>>>> > > >>>>>>> > > Once those requirements are settled, we can evaluate the >>>>>>> candidate >>>>>>> > > representations against the same semantics using reproducible >>>>>>> > > benchmarks. >>>>>>> > > >>>>>>> > > Best, >>>>>>> > > Costas Zarifis >>>>>>> > > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > THOMAS KISSINGER >>>>>>> > Staff Software Engineer >>>>>>> > >>>>>>> > MOBILE +49 174-2195270 <+49%20174%202195270> >>>>>>> <+49%20174%202195270> >>>>>>> > EMAIL [email protected] >>>>>>> > >>>>>>> > >>>>>>> > Snowflake Inc. >>>>>>> > 135 Constitution Drive >>>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>>> >>>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g>> >>>>>>> Menlo Park, CA 94025, USA >>>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>>> >>>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>>> > >>>>>>> >>>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>> >>>>>> >>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>> >>>>> >>>>> >>>>> -- >>>>> THOMAS KISSINGER >>>>> Staff Software Engineer >>>>> >>>>> MOBILE +49 174-2195270 <+49%20174%202195270> >>>>> EMAIL [email protected] >>>>> >>>>> >>>>> Snowflake Inc. >>>>> 135 Constitution Drive >>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>> Menlo Park, CA 94025, USA >>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>> >>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>> >>>> >>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>> >>> >>> >>> -- >>> THOMAS KISSINGER >>> Staff Software Engineer >>> >>> MOBILE +49 174-2195270 <+49%20174%202195270> >>> EMAIL [email protected] >>> >>> >>> Snowflake Inc. >>> 135 Constitution Drive >>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>> Menlo Park, CA 94025, USA >>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>> >> -- THOMAS KISSINGER Staff Software Engineer MOBILE +49 174-2195270 EMAIL [email protected] Snowflake Inc. 135 Constitution Drive Menlo Park, CA 94025, USA
