IN my mind, the fact that "distinct" is not precisely defined for all types means that changing "count of distinct values occurring" to "Estimated count of distinct values occurring." is a clarification
It seems like a significant change in meaning (explicitly allowing estimates) to me <shrug> Andrew On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> wrote: > I think the issue is less about whether this is changing something for > older readers/writers or providing a more accurate description. > > Older readers would already "misinterpret" written values because there's a > difference between logical equivalence and physical equivalence. This > difference is left up to interpretation by both the reader and writer. > > An example would be -0 vs +0 for floating point types where some systems > treat them equally where others may distinguish them. Similarly NaN may or > may not be included as a value count. > > I think it's wrong to assume that current readers/writers are capturing > this "correctly" when there is no "correct" or agreedupon representation. > > -Dan > > On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer <[email protected] > > > wrote: > > > My understanding is that the exactness of NDV is not standardized at all > > for many types, which makes this value very difficult to define or > > standardize as is. The main callouts during the meeting were strings with > > collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 etc...) So > > since we don't want to spend time addressing these issues, as most use > > cases only require approximate statistics, the consensus was to correct > the > > language now rather than spend time establishing something new. Just a > > reminder for everyone, please make sure to add notes when referring to > > discussions that occurred off list. I know the full video and trascript > > will eventually be available but it's helpful to keep things straight :) > > From the notes doc - > > > > - > > > > exactness of NDV stats: > > - > > > > Wanted to use in datafusion > > - > > > > Question of exactitude. > > - > > > > Collations makes it difficult because two distinct values might be > > considered equal. > > - > > > > Semantics are under defined. > > - > > > > Do we want to formalize for floating point numbers: -0, +0, > NaNs, > > etc? > > - > > > > Adrian: > > - > > > > Define right now as exact. > > - > > > > Most use cases do not require exact. > > - > > > > Statistical approximations of counts? > > - > > > > Kevin: > > - > > > > do we need to update the doc as well? > > - > > > > Yes! > > > > > > On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield <[email protected]> > > wrote: > > > > > Hi Adrian, > > > Is there more context from the call, on why Andrew's assessment didn't > > make > > > sense? Are there reference implementations providing approximate > answers > > > today? > > > > > > Thanks, > > > Micah > > > > > > On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > > > [email protected]> > > > wrote: > > > > > > > Per discussion on the sync call today I opened a format PR: > > > > https://github.com/apache/parquet-format/pull/628 > > > > > > > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > > > > I think optional approximate distinct (or other sketches of the > data > > > like > > > > > histograms) would help various database planners make better plans, > > so > > > > they > > > > > are valuable to explore. > > > > > > > > > > However, I think redefining the existing distinct_count statistic > > > > > would have all the forward compatibility problems we have been > > debating > > > > -- > > > > > namely we have no mechanism to prevent older readers > misinterpreting > > a > > > > > field whose semantic meaning was redefined. > > > > > > > > > > So maybe we could investigate adding new standardized metadata > > instead > > > of > > > > > redefining the meaning of an old one > > > > > > > > > > Andrew > > > > > > > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev < > > > > > [email protected]> wrote: > > > > > > > > > > > The specification does not define what > `Statistics.distinct_count` > > > > > > contains. Writers do not know if they can write an estimate. > > Readers > > > > do not > > > > > > know if they can use the value as an exact count. > > > > > > > > > > > > The field has this definition in `parquet.thrift`: > > > > > > > > > > > > ```thrift > > > > > > /** count of distinct values occurring */ > > > > > > 4: optional i64 distinct_count; > > > > > > ``` > > > > > > > > > > > > The text does not answer these questions: > > > > > > > > > > > > 1. Must the value be exact? Or can it be an estimate, for example > > > from > > > > a > > > > > > HyperLogLog sketch? > > > > > > > > > > > > If we choose to make the value inexact, this opens up the > question: > > > > > > > > > > > > 1. If the value is not exact, what does it guarantee? Is it an > > upper > > > > > > bound, a lower bound, or an estimate with no bound? > > > > > > > > > > > > If we choose to make the value exact, this opens up other > > questions: > > > > > > > > > > > > 1. How do nulls factor into the count? > > > > > > 2. How do writers compare values? For example, are `-0.0` and > > `+0.0` > > > > one > > > > > > value or two? Are two NaN values with different bit patterns one > > > value > > > > or > > > > > > two? Is the comparison on the physical bytes or on the logical > > type? > > > > > > 3. What does the value count for nested columns? (Relevant: > > > > > > https://github.com/apache/parquet-format/issues/476) > > > > > > > > > > > > It seems to me that an exact count per row group is not very > > useful: > > > > since > > > > > > it cannot be combined across row groups or files any sort of > > > > optimization > > > > > > that attempts to use it for a query like `select count(distinct > > col) > > > > from > > > > > > t` breaks for many queries (also any query with a filter, not to > > > > mention > > > > > > group by, etc.). We could do what Iceberg seems to do and store > > > > sketches > > > > > > that can be combined across files/row groups, but I think that > > should > > > > be a > > > > > > separate discussion as it would involve new fields. > > > > > > > > > > > > Where this does seem like it could be useful is optimizers (e.g. > > join > > > > > > ordering). For those applications I think an inexact value is > just > > as > > > > good > > > > > > as an exact one. > > > > > > > > > > > > Thus I would like to propose that we clarify the spec to mark > this > > > > field > > > > > > as an estimate that can be off in either direction by an > unbounded > > > > amount. > > > > > > If there are good use cases for error estimates we could add that > > > > later, > > > > > > and if we do we can get back exact values by setting the error > to 0 > > > > > > explicitly. The change would be a (possibly) 1 line change in the > > > > comment > > > > > > of the thrift definition above. > > > > > > > > > > > > Below is some investigation I did with an agent to come to this > > > > conclusion. > > > > > > > > > > > > ## History > > > > > > > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > > > > distinct_values`. > > > > > > This was a list of all distinct values and the count of each > value. > > > > > > - 2013 (commit `d9039d0`): The field changed to `i64 > > distinct_count`. > > > > The > > > > > > commit message is "distinct values can be fetched from the > > dictionary > > > > > > page". This shows that the original design was an exact count of > > the > > > > > > dictionary values. But the specification text did not say this. > > > > > > - 2014 (commit `b2836e5`): The comment changed to the current > text. > > > > There > > > > > > is no change in meaning since then. > > > > > > - 2023 (PARQUET-2352, #216): The specification changed > `min_value` > > > and > > > > > > `max_value` from exact values to "lower and upper bound values". > It > > > > also > > > > > > added `is_min_value_exact` and `is_max_value_exact`. This change > > did > > > > not > > > > > > include `distinct_count`. > > > > > > - 2024: A message on [email protected] asked if > > > `distinct_count` > > > > is > > > > > > exact, and if there are use cases for it. The thread discussed > > > > `null_count` > > > > > > only. There was no answer about `distinct_count`. The result was > > > #449, > > > > > > which also changed `null_count` only. Thread: > > > > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > > > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min > > sketches > > > as > > > > > > new statistics. This proposal is not active. > > > > > > > > > > > > ## Current implementations > > > > > > > > > > > > Few implementations write this field. No implementation that we > > > > examined > > > > > > writes an estimate. There are known requests to write estimates, > > > > because an > > > > > > exact count for one row group has limited use for query planning > > (for > > > > > > example, https://github.com/apache/arrow-rs/issues/8608). > > > > > > > > > > > > ### Writers > > > > > > > > > > > > | Implementation | Writes `distinct_count`? | Value | > > > > > > |---|---|---| > > > > > > | parquet-java | No | Not applicable | > > > > > > | Arrow C++ | Only if the user supplies a value | The value from > > the > > > > user | > > > > > > | arrow-go | Only if the user supplies a value | The value from > the > > > > user | > > > > > > | arrow-rs | Optional, off by default (since 2026-08) | Intended > as > > > > exact. > > > > > > It counts 64-bit hashes of the non-null values in the row group. > | > > > > > > | DuckDB | Yes, when the column chunk is dictionary-encoded | The > > > > > > dictionary size | > > > > > > | Impala, cuDF, Polars | No | Not applicable | > > > > > > > > > > > > ### Readers > > > > > > > > > > > > | Implementation | Reads `distinct_count`? | Use | > > > > > > |---|---|---| > > > > > > | parquet-java and engines that use it (Spark, Trino, Presto, > Hive, > > > > Drill) > > > > > > | No | Not applicable | > > > > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only | > > > > > > | Polars | Yes | Cardinality estimation. It accepts an > approximate > > > > value. | > > > > > > | DataFusion | Yes | Cardinality estimation. If a file has one > row > > > > group, > > > > > > it also uses the value as an exact answer for `COUNT(DISTINCT > > col)`. > > > | > > > > > > > > > > > > ### Comparison: Apache Iceberg > > > > > > > > > > > > Iceberg keeps distinct-value statistics in Puffin files as Apache > > > > > > DataSketches theta sketches. The Puffin specification says that > the > > > > `ndv` > > > > > > property is an "estimate of number of distinct values, derived > from > > > the > > > > > > sketch". Iceberg keeps these estimates separate from its exact > file > > > > > > statistics. > > > > > > > > > > > > ## Possible solutions > > > > > > > > > > > > - Option A (exact): State that distinct_count MUST be the exact > > > number > > > > of > > > > > > distinct values. > > > > > > - Option B (estimate): State that distinct_count is an estimate, > > > which > > > > can > > > > > > be higher or lower than the actual number by an amount that the > > > > > > specification does not limit. > > > > > > - Option C (flag): Add an is_distinct_count_exact flag, similar > to > > > > > > is_min_value_exact and is_max_value_exact. > > > > > > Extension to Options B and C (error bounds): Add optional fields > > for > > > a > > > > > > lower bound, an upper bound and the confidence level of the > bounds. > > > > > > - Option D (separate field): Keep distinct_count exact and add a > > new > > > > field > > > > > > for an estimate or for a sketch that readers can merge, for > example > > > the > > > > > > Apache DataSketches format that Iceberg uses. > > > > > > - Option E (deprecate): Deprecate the field. > > > > > > > > > > > > > > >
