Hi Adrian, Is there more context from the call, on why Andrew's assessment didn't make sense? Are there reference implementations providing approximate answers today?
Thanks, Micah On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco <[email protected]> wrote: > Per discussion on the sync call today I opened a format PR: > https://github.com/apache/parquet-format/pull/628 > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > I think optional approximate distinct (or other sketches of the data like > > histograms) would help various database planners make better plans, so > they > > are valuable to explore. > > > > However, I think redefining the existing distinct_count statistic > > would have all the forward compatibility problems we have been debating > -- > > namely we have no mechanism to prevent older readers misinterpreting a > > field whose semantic meaning was redefined. > > > > So maybe we could investigate adding new standardized metadata instead of > > redefining the meaning of an old one > > > > Andrew > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev < > > [email protected]> wrote: > > > > > The specification does not define what `Statistics.distinct_count` > > > contains. Writers do not know if they can write an estimate. Readers > do not > > > know if they can use the value as an exact count. > > > > > > The field has this definition in `parquet.thrift`: > > > > > > ```thrift > > > /** count of distinct values occurring */ > > > 4: optional i64 distinct_count; > > > ``` > > > > > > The text does not answer these questions: > > > > > > 1. Must the value be exact? Or can it be an estimate, for example from > a > > > HyperLogLog sketch? > > > > > > If we choose to make the value inexact, this opens up the question: > > > > > > 1. If the value is not exact, what does it guarantee? Is it an upper > > > bound, a lower bound, or an estimate with no bound? > > > > > > If we choose to make the value exact, this opens up other questions: > > > > > > 1. How do nulls factor into the count? > > > 2. How do writers compare values? For example, are `-0.0` and `+0.0` > one > > > value or two? Are two NaN values with different bit patterns one value > or > > > two? Is the comparison on the physical bytes or on the logical type? > > > 3. What does the value count for nested columns? (Relevant: > > > https://github.com/apache/parquet-format/issues/476) > > > > > > It seems to me that an exact count per row group is not very useful: > since > > > it cannot be combined across row groups or files any sort of > optimization > > > that attempts to use it for a query like `select count(distinct col) > from > > > t` breaks for many queries (also any query with a filter, not to > mention > > > group by, etc.). We could do what Iceberg seems to do and store > sketches > > > that can be combined across files/row groups, but I think that should > be a > > > separate discussion as it would involve new fields. > > > > > > Where this does seem like it could be useful is optimizers (e.g. join > > > ordering). For those applications I think an inexact value is just as > good > > > as an exact one. > > > > > > Thus I would like to propose that we clarify the spec to mark this > field > > > as an estimate that can be off in either direction by an unbounded > amount. > > > If there are good use cases for error estimates we could add that > later, > > > and if we do we can get back exact values by setting the error to 0 > > > explicitly. The change would be a (possibly) 1 line change in the > comment > > > of the thrift definition above. > > > > > > Below is some investigation I did with an agent to come to this > conclusion. > > > > > > ## History > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > distinct_values`. > > > This was a list of all distinct values and the count of each value. > > > - 2013 (commit `d9039d0`): The field changed to `i64 distinct_count`. > The > > > commit message is "distinct values can be fetched from the dictionary > > > page". This shows that the original design was an exact count of the > > > dictionary values. But the specification text did not say this. > > > - 2014 (commit `b2836e5`): The comment changed to the current text. > There > > > is no change in meaning since then. > > > - 2023 (PARQUET-2352, #216): The specification changed `min_value` and > > > `max_value` from exact values to "lower and upper bound values". It > also > > > added `is_min_value_exact` and `is_max_value_exact`. This change did > not > > > include `distinct_count`. > > > - 2024: A message on [email protected] asked if `distinct_count` > is > > > exact, and if there are use cases for it. The thread discussed > `null_count` > > > only. There was no answer about `distinct_count`. The result was #449, > > > which also changed `null_count` only. Thread: > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min sketches as > > > new statistics. This proposal is not active. > > > > > > ## Current implementations > > > > > > Few implementations write this field. No implementation that we > examined > > > writes an estimate. There are known requests to write estimates, > because an > > > exact count for one row group has limited use for query planning (for > > > example, https://github.com/apache/arrow-rs/issues/8608). > > > > > > ### Writers > > > > > > | Implementation | Writes `distinct_count`? | Value | > > > |---|---|---| > > > | parquet-java | No | Not applicable | > > > | Arrow C++ | Only if the user supplies a value | The value from the > user | > > > | arrow-go | Only if the user supplies a value | The value from the > user | > > > | arrow-rs | Optional, off by default (since 2026-08) | Intended as > exact. > > > It counts 64-bit hashes of the non-null values in the row group. | > > > | DuckDB | Yes, when the column chunk is dictionary-encoded | The > > > dictionary size | > > > | Impala, cuDF, Polars | No | Not applicable | > > > > > > ### Readers > > > > > > | Implementation | Reads `distinct_count`? | Use | > > > |---|---|---| > > > | parquet-java and engines that use it (Spark, Trino, Presto, Hive, > Drill) > > > | No | Not applicable | > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only | > > > | Polars | Yes | Cardinality estimation. It accepts an approximate > value. | > > > | DataFusion | Yes | Cardinality estimation. If a file has one row > group, > > > it also uses the value as an exact answer for `COUNT(DISTINCT col)`. | > > > > > > ### Comparison: Apache Iceberg > > > > > > Iceberg keeps distinct-value statistics in Puffin files as Apache > > > DataSketches theta sketches. The Puffin specification says that the > `ndv` > > > property is an "estimate of number of distinct values, derived from the > > > sketch". Iceberg keeps these estimates separate from its exact file > > > statistics. > > > > > > ## Possible solutions > > > > > > - Option A (exact): State that distinct_count MUST be the exact number > of > > > distinct values. > > > - Option B (estimate): State that distinct_count is an estimate, which > can > > > be higher or lower than the actual number by an amount that the > > > specification does not limit. > > > - Option C (flag): Add an is_distinct_count_exact flag, similar to > > > is_min_value_exact and is_max_value_exact. > > > Extension to Options B and C (error bounds): Add optional fields for a > > > lower bound, an upper bound and the confidence level of the bounds. > > > - Option D (separate field): Keep distinct_count exact and add a new > field > > > for an estimate or for a sketch that readers can merge, for example the > > > Apache DataSketches format that Iceberg uses. > > > - Option E (deprecate): Deprecate the field. > > >
