I think the issue is less about whether this is changing something for older readers/writers or providing a more accurate description.
Older readers would already "misinterpret" written values because there's a difference between logical equivalence and physical equivalence. This difference is left up to interpretation by both the reader and writer. An example would be -0 vs +0 for floating point types where some systems treat them equally where others may distinguish them. Similarly NaN may or may not be included as a value count. I think it's wrong to assume that current readers/writers are capturing this "correctly" when there is no "correct" or agreedupon representation. -Dan On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer <[email protected]> wrote: > My understanding is that the exactness of NDV is not standardized at all > for many types, which makes this value very difficult to define or > standardize as is. The main callouts during the meeting were strings with > collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 etc...) So > since we don't want to spend time addressing these issues, as most use > cases only require approximate statistics, the consensus was to correct the > language now rather than spend time establishing something new. Just a > reminder for everyone, please make sure to add notes when referring to > discussions that occurred off list. I know the full video and trascript > will eventually be available but it's helpful to keep things straight :) > From the notes doc - > > - > > exactness of NDV stats: > - > > Wanted to use in datafusion > - > > Question of exactitude. > - > > Collations makes it difficult because two distinct values might be > considered equal. > - > > Semantics are under defined. > - > > Do we want to formalize for floating point numbers: -0, +0, NaNs, > etc? > - > > Adrian: > - > > Define right now as exact. > - > > Most use cases do not require exact. > - > > Statistical approximations of counts? > - > > Kevin: > - > > do we need to update the doc as well? > - > > Yes! > > > On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield <[email protected]> > wrote: > > > Hi Adrian, > > Is there more context from the call, on why Andrew's assessment didn't > make > > sense? Are there reference implementations providing approximate answers > > today? > > > > Thanks, > > Micah > > > > On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > > [email protected]> > > wrote: > > > > > Per discussion on the sync call today I opened a format PR: > > > https://github.com/apache/parquet-format/pull/628 > > > > > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > > > I think optional approximate distinct (or other sketches of the data > > like > > > > histograms) would help various database planners make better plans, > so > > > they > > > > are valuable to explore. > > > > > > > > However, I think redefining the existing distinct_count statistic > > > > would have all the forward compatibility problems we have been > debating > > > -- > > > > namely we have no mechanism to prevent older readers misinterpreting > a > > > > field whose semantic meaning was redefined. > > > > > > > > So maybe we could investigate adding new standardized metadata > instead > > of > > > > redefining the meaning of an old one > > > > > > > > Andrew > > > > > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev < > > > > [email protected]> wrote: > > > > > > > > > The specification does not define what `Statistics.distinct_count` > > > > > contains. Writers do not know if they can write an estimate. > Readers > > > do not > > > > > know if they can use the value as an exact count. > > > > > > > > > > The field has this definition in `parquet.thrift`: > > > > > > > > > > ```thrift > > > > > /** count of distinct values occurring */ > > > > > 4: optional i64 distinct_count; > > > > > ``` > > > > > > > > > > The text does not answer these questions: > > > > > > > > > > 1. Must the value be exact? Or can it be an estimate, for example > > from > > > a > > > > > HyperLogLog sketch? > > > > > > > > > > If we choose to make the value inexact, this opens up the question: > > > > > > > > > > 1. If the value is not exact, what does it guarantee? Is it an > upper > > > > > bound, a lower bound, or an estimate with no bound? > > > > > > > > > > If we choose to make the value exact, this opens up other > questions: > > > > > > > > > > 1. How do nulls factor into the count? > > > > > 2. How do writers compare values? For example, are `-0.0` and > `+0.0` > > > one > > > > > value or two? Are two NaN values with different bit patterns one > > value > > > or > > > > > two? Is the comparison on the physical bytes or on the logical > type? > > > > > 3. What does the value count for nested columns? (Relevant: > > > > > https://github.com/apache/parquet-format/issues/476) > > > > > > > > > > It seems to me that an exact count per row group is not very > useful: > > > since > > > > > it cannot be combined across row groups or files any sort of > > > optimization > > > > > that attempts to use it for a query like `select count(distinct > col) > > > from > > > > > t` breaks for many queries (also any query with a filter, not to > > > mention > > > > > group by, etc.). We could do what Iceberg seems to do and store > > > sketches > > > > > that can be combined across files/row groups, but I think that > should > > > be a > > > > > separate discussion as it would involve new fields. > > > > > > > > > > Where this does seem like it could be useful is optimizers (e.g. > join > > > > > ordering). For those applications I think an inexact value is just > as > > > good > > > > > as an exact one. > > > > > > > > > > Thus I would like to propose that we clarify the spec to mark this > > > field > > > > > as an estimate that can be off in either direction by an unbounded > > > amount. > > > > > If there are good use cases for error estimates we could add that > > > later, > > > > > and if we do we can get back exact values by setting the error to 0 > > > > > explicitly. The change would be a (possibly) 1 line change in the > > > comment > > > > > of the thrift definition above. > > > > > > > > > > Below is some investigation I did with an agent to come to this > > > conclusion. > > > > > > > > > > ## History > > > > > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > > > distinct_values`. > > > > > This was a list of all distinct values and the count of each value. > > > > > - 2013 (commit `d9039d0`): The field changed to `i64 > distinct_count`. > > > The > > > > > commit message is "distinct values can be fetched from the > dictionary > > > > > page". This shows that the original design was an exact count of > the > > > > > dictionary values. But the specification text did not say this. > > > > > - 2014 (commit `b2836e5`): The comment changed to the current text. > > > There > > > > > is no change in meaning since then. > > > > > - 2023 (PARQUET-2352, #216): The specification changed `min_value` > > and > > > > > `max_value` from exact values to "lower and upper bound values". It > > > also > > > > > added `is_min_value_exact` and `is_max_value_exact`. This change > did > > > not > > > > > include `distinct_count`. > > > > > - 2024: A message on [email protected] asked if > > `distinct_count` > > > is > > > > > exact, and if there are use cases for it. The thread discussed > > > `null_count` > > > > > only. There was no answer about `distinct_count`. The result was > > #449, > > > > > which also changed `null_count` only. Thread: > > > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min > sketches > > as > > > > > new statistics. This proposal is not active. > > > > > > > > > > ## Current implementations > > > > > > > > > > Few implementations write this field. No implementation that we > > > examined > > > > > writes an estimate. There are known requests to write estimates, > > > because an > > > > > exact count for one row group has limited use for query planning > (for > > > > > example, https://github.com/apache/arrow-rs/issues/8608). > > > > > > > > > > ### Writers > > > > > > > > > > | Implementation | Writes `distinct_count`? | Value | > > > > > |---|---|---| > > > > > | parquet-java | No | Not applicable | > > > > > | Arrow C++ | Only if the user supplies a value | The value from > the > > > user | > > > > > | arrow-go | Only if the user supplies a value | The value from the > > > user | > > > > > | arrow-rs | Optional, off by default (since 2026-08) | Intended as > > > exact. > > > > > It counts 64-bit hashes of the non-null values in the row group. | > > > > > | DuckDB | Yes, when the column chunk is dictionary-encoded | The > > > > > dictionary size | > > > > > | Impala, cuDF, Polars | No | Not applicable | > > > > > > > > > > ### Readers > > > > > > > > > > | Implementation | Reads `distinct_count`? | Use | > > > > > |---|---|---| > > > > > | parquet-java and engines that use it (Spark, Trino, Presto, Hive, > > > Drill) > > > > > | No | Not applicable | > > > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only | > > > > > | Polars | Yes | Cardinality estimation. It accepts an approximate > > > value. | > > > > > | DataFusion | Yes | Cardinality estimation. If a file has one row > > > group, > > > > > it also uses the value as an exact answer for `COUNT(DISTINCT > col)`. > > | > > > > > > > > > > ### Comparison: Apache Iceberg > > > > > > > > > > Iceberg keeps distinct-value statistics in Puffin files as Apache > > > > > DataSketches theta sketches. The Puffin specification says that the > > > `ndv` > > > > > property is an "estimate of number of distinct values, derived from > > the > > > > > sketch". Iceberg keeps these estimates separate from its exact file > > > > > statistics. > > > > > > > > > > ## Possible solutions > > > > > > > > > > - Option A (exact): State that distinct_count MUST be the exact > > number > > > of > > > > > distinct values. > > > > > - Option B (estimate): State that distinct_count is an estimate, > > which > > > can > > > > > be higher or lower than the actual number by an amount that the > > > > > specification does not limit. > > > > > - Option C (flag): Add an is_distinct_count_exact flag, similar to > > > > > is_min_value_exact and is_max_value_exact. > > > > > Extension to Options B and C (error bounds): Add optional fields > for > > a > > > > > lower bound, an upper bound and the confidence level of the bounds. > > > > > - Option D (separate field): Keep distinct_count exact and add a > new > > > field > > > > > for an estimate or for a sketch that readers can merge, for example > > the > > > > > Apache DataSketches format that Iceberg uses. > > > > > - Option E (deprecate): Deprecate the field. > > > > > > > > > >
