Per discussion on the sync call today I opened a format PR: https://github.com/apache/parquet-format/pull/628
On 2026/09/30 21:26:04 Andrew Lamb wrote: > I think optional approximate distinct (or other sketches of the data like > histograms) would help various database planners make better plans, so they > are valuable to explore. > > However, I think redefining the existing distinct_count statistic > would have all the forward compatibility problems we have been debating -- > namely we have no mechanism to prevent older readers misinterpreting a > field whose semantic meaning was redefined. > > So maybe we could investigate adding new standardized metadata instead of > redefining the meaning of an old one > > Andrew > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev < > [email protected]> wrote: > > > The specification does not define what `Statistics.distinct_count` > > contains. Writers do not know if they can write an estimate. Readers do not > > know if they can use the value as an exact count. > > > > The field has this definition in `parquet.thrift`: > > > > ```thrift > > /** count of distinct values occurring */ > > 4: optional i64 distinct_count; > > ``` > > > > The text does not answer these questions: > > > > 1. Must the value be exact? Or can it be an estimate, for example from a > > HyperLogLog sketch? > > > > If we choose to make the value inexact, this opens up the question: > > > > 1. If the value is not exact, what does it guarantee? Is it an upper > > bound, a lower bound, or an estimate with no bound? > > > > If we choose to make the value exact, this opens up other questions: > > > > 1. How do nulls factor into the count? > > 2. How do writers compare values? For example, are `-0.0` and `+0.0` one > > value or two? Are two NaN values with different bit patterns one value or > > two? Is the comparison on the physical bytes or on the logical type? > > 3. What does the value count for nested columns? (Relevant: > > https://github.com/apache/parquet-format/issues/476) > > > > It seems to me that an exact count per row group is not very useful: since > > it cannot be combined across row groups or files any sort of optimization > > that attempts to use it for a query like `select count(distinct col) from > > t` breaks for many queries (also any query with a filter, not to mention > > group by, etc.). We could do what Iceberg seems to do and store sketches > > that can be combined across files/row groups, but I think that should be a > > separate discussion as it would involve new fields. > > > > Where this does seem like it could be useful is optimizers (e.g. join > > ordering). For those applications I think an inexact value is just as good > > as an exact one. > > > > Thus I would like to propose that we clarify the spec to mark this field > > as an estimate that can be off in either direction by an unbounded amount. > > If there are good use cases for error estimates we could add that later, > > and if we do we can get back exact values by setting the error to 0 > > explicitly. The change would be a (possibly) 1 line change in the comment > > of the thrift definition above. > > > > Below is some investigation I did with an agent to come to this conclusion. > > > > ## History > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> distinct_values`. > > This was a list of all distinct values and the count of each value. > > - 2013 (commit `d9039d0`): The field changed to `i64 distinct_count`. The > > commit message is "distinct values can be fetched from the dictionary > > page". This shows that the original design was an exact count of the > > dictionary values. But the specification text did not say this. > > - 2014 (commit `b2836e5`): The comment changed to the current text. There > > is no change in meaning since then. > > - 2023 (PARQUET-2352, #216): The specification changed `min_value` and > > `max_value` from exact values to "lower and upper bound values". It also > > added `is_min_value_exact` and `is_max_value_exact`. This change did not > > include `distinct_count`. > > - 2024: A message on [email protected] asked if `distinct_count` is > > exact, and if there are use cases for it. The thread discussed `null_count` > > only. There was no answer about `distinct_count`. The result was #449, > > which also changed `null_count` only. Thread: > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min sketches as > > new statistics. This proposal is not active. > > > > ## Current implementations > > > > Few implementations write this field. No implementation that we examined > > writes an estimate. There are known requests to write estimates, because an > > exact count for one row group has limited use for query planning (for > > example, https://github.com/apache/arrow-rs/issues/8608). > > > > ### Writers > > > > | Implementation | Writes `distinct_count`? | Value | > > |---|---|---| > > | parquet-java | No | Not applicable | > > | Arrow C++ | Only if the user supplies a value | The value from the user | > > | arrow-go | Only if the user supplies a value | The value from the user | > > | arrow-rs | Optional, off by default (since 2026-08) | Intended as exact. > > It counts 64-bit hashes of the non-null values in the row group. | > > | DuckDB | Yes, when the column chunk is dictionary-encoded | The > > dictionary size | > > | Impala, cuDF, Polars | No | Not applicable | > > > > ### Readers > > > > | Implementation | Reads `distinct_count`? | Use | > > |---|---|---| > > | parquet-java and engines that use it (Spark, Trino, Presto, Hive, Drill) > > | No | Not applicable | > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only | > > | Polars | Yes | Cardinality estimation. It accepts an approximate value. | > > | DataFusion | Yes | Cardinality estimation. If a file has one row group, > > it also uses the value as an exact answer for `COUNT(DISTINCT col)`. | > > > > ### Comparison: Apache Iceberg > > > > Iceberg keeps distinct-value statistics in Puffin files as Apache > > DataSketches theta sketches. The Puffin specification says that the `ndv` > > property is an "estimate of number of distinct values, derived from the > > sketch". Iceberg keeps these estimates separate from its exact file > > statistics. > > > > ## Possible solutions > > > > - Option A (exact): State that distinct_count MUST be the exact number of > > distinct values. > > - Option B (estimate): State that distinct_count is an estimate, which can > > be higher or lower than the actual number by an amount that the > > specification does not limit. > > - Option C (flag): Add an is_distinct_count_exact flag, similar to > > is_min_value_exact and is_max_value_exact. > > Extension to Options B and C (error bounds): Add optional fields for a > > lower bound, an upper bound and the confidence level of the bounds. > > - Option D (separate field): Keep distinct_count exact and add a new field > > for an estimate or for a sketch that readers can merge, for example the > > Apache DataSketches format that Iceberg uses. > > - Option E (deprecate): Deprecate the field. >
