mapleFU commented on code in PR #40594:
URL: https://github.com/apache/arrow/pull/40594#discussion_r1856704315
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -1554,6 +1591,46 @@ class TypedColumnWriterImpl : public ColumnWriterImpl,
public TypedColumnWriter<
rows_written_ += num_levels;
num_buffered_rows_ += num_levels;
}
+
+ UpdateLevelHistogram(num_levels, def_levels, rep_levels);
+ }
+
+ void UpdateLevelHistogram(int64_t num_levels, const int16_t* def_levels,
+ const int16_t* rep_levels) const {
+ if (page_size_statistics_ == nullptr) {
+ return;
+ }
+
+ auto add_levels = [](std::vector<int64_t>& level_histogram,
+ ::arrow::util::span<const int16_t> levels) {
Review Comment:
Mixing vector and span<const> is a bit weird...
##########
cpp/src/parquet/encoder.cc:
##########
@@ -633,6 +669,7 @@ inline void DictEncoderImpl<DType>::Put(const T& v) {
int32_t memo_index;
PARQUET_THROW_NOT_OK(memo_table_.GetOrInsert(v, on_found, on_not_found,
&memo_index));
buffered_indices_.push_back(memo_index);
+ unencoded_data_bytes_ += sizeof(T);
Review Comment:
I'm a little suspect the idea here, since for non-byte-array type, these can
just be caculated by the `buffered_indices_`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
