pitrou commented on code in PR #35989:
URL: https://github.com/apache/arrow/pull/35989#discussion_r1242395214
##########
cpp/src/parquet/statistics.cc:
##########
@@ -610,8 +619,12 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
if (HasNullCount()) {
s.set_null_count(this->null_count());
}
- // num_values_ is reliable and it means number of non-null values.
- s.all_null_value = num_values_ == 0;
+ if (HasNullCount()) {
+ // num_values_ is reliable and it means number of non-null values.
+ s.all_null_value = num_values_ == 0;
Review Comment:
Can you add a statistics test for `all_null_value`? Including cases where
`num_values_` is not reliable.
##########
cpp/src/parquet/statistics.cc:
##########
@@ -610,8 +619,12 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
if (HasNullCount()) {
s.set_null_count(this->null_count());
}
- // num_values_ is reliable and it means number of non-null values.
- s.all_null_value = num_values_ == 0;
+ if (HasNullCount()) {
Review Comment:
Can you merge this with the `if` before?
##########
cpp/src/parquet/statistics.cc:
##########
@@ -627,7 +640,12 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
T min_;
T max_;
::arrow::MemoryPool* pool_;
- int64_t num_values_ = 0; // # of non-null values.
+ // # of non-null values.
+ // num_values_ would be reliable when has_null_count_,
+ // if it's created from a page thrift statistics, and statistics
+ // doesn't have null_count, but page has null data, `num_values_`
+ // would be equal to value including nulls.
Review Comment:
This isn't very readable. Do you want to split this into two sentences?
##########
cpp/src/parquet/statistics.cc:
##########
@@ -637,8 +655,9 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
void Copy(const T& src, T* dst, ResizableBuffer*) { *dst = src; }
- void IncrementDistinctCount(int64_t n) {
- statistics_.distinct_count += n;
+ void SetDistinctCount(int64_t n) {
+ // distinct count can only be "set", and cannot be increment.
Review Comment:
```suggestion
// distinct count can only be "set", and cannot be incremented.
```
##########
cpp/src/parquet/statistics.cc:
##########
@@ -627,7 +640,12 @@ class TypedStatisticsImpl : public TypedStatistics<DType> {
T min_;
T max_;
::arrow::MemoryPool* pool_;
- int64_t num_values_ = 0; // # of non-null values.
+ // # of non-null values.
Review Comment:
```suggestion
// Number of non-null values.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
