My understanding is that the exactness of NDV is not standardized at all
for many types, which makes this value very difficult to define or
standardize as is. The main callouts during the meeting were strings with
collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 etc...) So
since we don't want to spend time addressing these issues, as most use
cases only require approximate statistics, the consensus was to correct the
language now rather than spend time establishing something new. Just a
reminder for everyone, please make sure to add notes when referring to
discussions that occurred off list. I know the full video and trascript
will eventually be available but it's helpful to keep things straight :)
>From the notes doc -
-
exactness of NDV stats:
-
Wanted to use in datafusion
-
Question of exactitude.
-
Collations makes it difficult because two distinct values might be
considered equal.
-
Semantics are under defined.
-
Do we want to formalize for floating point numbers: -0, +0, NaNs,
etc?
-
Adrian:
-
Define right now as exact.
-
Most use cases do not require exact.
-
Statistical approximations of counts?
-
Kevin:
-
do we need to update the doc as well?
-
Yes!
On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield <[email protected]>
wrote:
> Hi Adrian,
> Is there more context from the call, on why Andrew's assessment didn't make
> sense? Are there reference implementations providing approximate answers
> today?
>
> Thanks,
> Micah
>
> On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco <
> [email protected]>
> wrote:
>
> > Per discussion on the sync call today I opened a format PR:
> > https://github.com/apache/parquet-format/pull/628
> >
> > On 2026/09/30 21:26:04 Andrew Lamb wrote:
> > > I think optional approximate distinct (or other sketches of the data
> like
> > > histograms) would help various database planners make better plans, so
> > they
> > > are valuable to explore.
> > >
> > > However, I think redefining the existing distinct_count statistic
> > > would have all the forward compatibility problems we have been debating
> > --
> > > namely we have no mechanism to prevent older readers misinterpreting a
> > > field whose semantic meaning was redefined.
> > >
> > > So maybe we could investigate adding new standardized metadata instead
> of
> > > redefining the meaning of an old one
> > >
> > > Andrew
> > >
> > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev <
> > > [email protected]> wrote:
> > >
> > > > The specification does not define what `Statistics.distinct_count`
> > > > contains. Writers do not know if they can write an estimate. Readers
> > do not
> > > > know if they can use the value as an exact count.
> > > >
> > > > The field has this definition in `parquet.thrift`:
> > > >
> > > > ```thrift
> > > > /** count of distinct values occurring */
> > > > 4: optional i64 distinct_count;
> > > > ```
> > > >
> > > > The text does not answer these questions:
> > > >
> > > > 1. Must the value be exact? Or can it be an estimate, for example
> from
> > a
> > > > HyperLogLog sketch?
> > > >
> > > > If we choose to make the value inexact, this opens up the question:
> > > >
> > > > 1. If the value is not exact, what does it guarantee? Is it an upper
> > > > bound, a lower bound, or an estimate with no bound?
> > > >
> > > > If we choose to make the value exact, this opens up other questions:
> > > >
> > > > 1. How do nulls factor into the count?
> > > > 2. How do writers compare values? For example, are `-0.0` and `+0.0`
> > one
> > > > value or two? Are two NaN values with different bit patterns one
> value
> > or
> > > > two? Is the comparison on the physical bytes or on the logical type?
> > > > 3. What does the value count for nested columns? (Relevant:
> > > > https://github.com/apache/parquet-format/issues/476)
> > > >
> > > > It seems to me that an exact count per row group is not very useful:
> > since
> > > > it cannot be combined across row groups or files any sort of
> > optimization
> > > > that attempts to use it for a query like `select count(distinct col)
> > from
> > > > t` breaks for many queries (also any query with a filter, not to
> > mention
> > > > group by, etc.). We could do what Iceberg seems to do and store
> > sketches
> > > > that can be combined across files/row groups, but I think that should
> > be a
> > > > separate discussion as it would involve new fields.
> > > >
> > > > Where this does seem like it could be useful is optimizers (e.g. join
> > > > ordering). For those applications I think an inexact value is just as
> > good
> > > > as an exact one.
> > > >
> > > > Thus I would like to propose that we clarify the spec to mark this
> > field
> > > > as an estimate that can be off in either direction by an unbounded
> > amount.
> > > > If there are good use cases for error estimates we could add that
> > later,
> > > > and if we do we can get back exact values by setting the error to 0
> > > > explicitly. The change would be a (possibly) 1 line change in the
> > comment
> > > > of the thrift definition above.
> > > >
> > > > Below is some investigation I did with an agent to come to this
> > conclusion.
> > > >
> > > > ## History
> > > >
> > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats>
> > distinct_values`.
> > > > This was a list of all distinct values and the count of each value.
> > > > - 2013 (commit `d9039d0`): The field changed to `i64 distinct_count`.
> > The
> > > > commit message is "distinct values can be fetched from the dictionary
> > > > page". This shows that the original design was an exact count of the
> > > > dictionary values. But the specification text did not say this.
> > > > - 2014 (commit `b2836e5`): The comment changed to the current text.
> > There
> > > > is no change in meaning since then.
> > > > - 2023 (PARQUET-2352, #216): The specification changed `min_value`
> and
> > > > `max_value` from exact values to "lower and upper bound values". It
> > also
> > > > added `is_min_value_exact` and `is_max_value_exact`. This change did
> > not
> > > > include `distinct_count`.
> > > > - 2024: A message on [email protected] asked if
> `distinct_count`
> > is
> > > > exact, and if there are use cases for it. The thread discussed
> > `null_count`
> > > > only. There was no answer about `distinct_count`. The result was
> #449,
> > > > which also changed `null_count` only. Thread:
> > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt
> > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min sketches
> as
> > > > new statistics. This proposal is not active.
> > > >
> > > > ## Current implementations
> > > >
> > > > Few implementations write this field. No implementation that we
> > examined
> > > > writes an estimate. There are known requests to write estimates,
> > because an
> > > > exact count for one row group has limited use for query planning (for
> > > > example, https://github.com/apache/arrow-rs/issues/8608).
> > > >
> > > > ### Writers
> > > >
> > > > | Implementation | Writes `distinct_count`? | Value |
> > > > |---|---|---|
> > > > | parquet-java | No | Not applicable |
> > > > | Arrow C++ | Only if the user supplies a value | The value from the
> > user |
> > > > | arrow-go | Only if the user supplies a value | The value from the
> > user |
> > > > | arrow-rs | Optional, off by default (since 2026-08) | Intended as
> > exact.
> > > > It counts 64-bit hashes of the non-null values in the row group. |
> > > > | DuckDB | Yes, when the column chunk is dictionary-encoded | The
> > > > dictionary size |
> > > > | Impala, cuDF, Polars | No | Not applicable |
> > > >
> > > > ### Readers
> > > >
> > > > | Implementation | Reads `distinct_count`? | Use |
> > > > |---|---|---|
> > > > | parquet-java and engines that use it (Spark, Trino, Presto, Hive,
> > Drill)
> > > > | No | Not applicable |
> > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only |
> > > > | Polars | Yes | Cardinality estimation. It accepts an approximate
> > value. |
> > > > | DataFusion | Yes | Cardinality estimation. If a file has one row
> > group,
> > > > it also uses the value as an exact answer for `COUNT(DISTINCT col)`.
> |
> > > >
> > > > ### Comparison: Apache Iceberg
> > > >
> > > > Iceberg keeps distinct-value statistics in Puffin files as Apache
> > > > DataSketches theta sketches. The Puffin specification says that the
> > `ndv`
> > > > property is an "estimate of number of distinct values, derived from
> the
> > > > sketch". Iceberg keeps these estimates separate from its exact file
> > > > statistics.
> > > >
> > > > ## Possible solutions
> > > >
> > > > - Option A (exact): State that distinct_count MUST be the exact
> number
> > of
> > > > distinct values.
> > > > - Option B (estimate): State that distinct_count is an estimate,
> which
> > can
> > > > be higher or lower than the actual number by an amount that the
> > > > specification does not limit.
> > > > - Option C (flag): Add an is_distinct_count_exact flag, similar to
> > > > is_min_value_exact and is_max_value_exact.
> > > > Extension to Options B and C (error bounds): Add optional fields for
> a
> > > > lower bound, an upper bound and the confidence level of the bounds.
> > > > - Option D (separate field): Keep distinct_count exact and add a new
> > field
> > > > for an estimate or for a sketch that readers can merge, for example
> the
> > > > Apache DataSketches format that Iceberg uses.
> > > > - Option E (deprecate): Deprecate the field.
> > >
> >
>
