divjotarora commented on code in PR #628:
URL: https://github.com/apache/parquet-format/pull/628#discussion_r4465335644
##########
src/main/thrift/parquet.thrift:
##########
@@ -289,7 +289,10 @@ struct Statistics {
* If null_count is not present, readers MUST NOT assume null_count == 0.
*/
3: optional i64 null_count;
- /** count of distinct values occurring */
+ /**
+ * Estimated count of distinct values occurring.
+ * The estimate may be off in either direction by an underfined amount.
Review Comment:
I prefer @danielcweeks suggested wording. @adriangb IMO what this buys us is
a clear and targeted fix saying that `distinct_count` is essentially not useful
_for types where equality is vague or has multiple interpretations_. But for
types that can be unambiguously compared (integers, strings), `distinct_count`
remains an exact count if set. The initially proposed wording ("Estimated count
of distinct values occurring...") is a much broader statement that's redefining
this field, which feels like a breaking change. I also agree with the
conversation on the mailing list that if estimate count is what we want, we
should add another field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]