divjotarora commented on code in PR #628:
URL: https://github.com/apache/parquet-format/pull/628#discussion_r4465335644


##########
src/main/thrift/parquet.thrift:
##########
@@ -289,7 +289,10 @@ struct Statistics {
     * If null_count is not present, readers MUST NOT assume null_count == 0.
     */
    3: optional i64 null_count;
-   /** count of distinct values occurring */
+   /** 
+    * Estimated count of distinct values occurring.
+    * The estimate may be off in either direction by an underfined amount. 

Review Comment:
   I prefer @danielcweeks suggested wording. @adriangb IMO what this buys us is 
a clear and targeted fix saying that `distinct_count` is essentially not useful 
_for types where equality is vague or has multiple interpretations_. But for 
types that can be unambiguously compared (integers, strings), `distinct_count` 
remains an exact count if set. The initially proposed wording ("Estimated count 
of distinct values occurring...") is a much broader statement that's redefining 
this field, which feels like a breaking change. I also agree with the 
conversation on the mailing list that if estimate count is what we want, we 
should add another field.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to