[
https://issues.apache.org/jira/browse/PARQUET-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706170#comment-17706170
]
ASF GitHub Bot commented on PARQUET-2261:
-----------------------------------------
wgtmac commented on code in PR #197:
URL: https://github.com/apache/parquet-format/pull/197#discussion_r1151301394
##########
src/main/thrift/parquet.thrift:
##########
@@ -190,6 +190,41 @@ enum FieldRepetitionType {
/** The field is repeated and can contain 0 or more values */
REPEATED = 2;
}
+/**
+ * A structure for capturing metadata for estimating the unencoded,
uncompressed size
+ * of data.
+ *
+ * Writers should populate all fields in this struct except for the exceptions
listed per field.
+ */
+struct SizeEstimationStatistics {
+ /**
+ * The number of logical physical bytes stored for BYTE_ARRAY data values.
Logical bytes refers to the number
+ * of bytes needed if no special encoding is used. This is exclusive of the
bytes needed
+ * to store the length of each byte array. In other words, this field is
equivelant to the the (size of
+ * PLAIN-ENCODING the byte array values) - (4 bytes * number of values
written). To determine logical sizes
+ * of other other types readers can use schema information multiplied by
the number of non-null values.
+ * The number of non-null values can be inferred from the histograms below.
+ *
+ * For example if column chunk is dictionary encoded with a dictionary
["a", "bc", "cde"] and a data page
+ * has indexes [0, 0, 1, 2]. This value is expected to be 7 (1 + 1 + 2 +
3).
+ *
+ * This option should only be set for physical and logical types that would
use BYTE_ARRAY when encoded with PLAIN encoding.
+ */
+ 1: optional i64 logical_variable_width_stored_bytes;
+ /**
+ * When present there is expected to be one element corresponding to each
repetition (i.e. size=max repetition_level+1)
+ * where each element represens the number of time the repetition level
was observed in the data.
+ *
+ * This value is optional if max_repetition_level is 0.
+ */
+ 2: optional list<i64> repetition_level_histogram;
+ /**
+ * Same as repetition_level_histogram except for definition levels.
+ *
+ * This value is optional when max_definition_level is 0.
+ */
+ 3: optional list<i64> definition_level_histogram;
Review Comment:
If we store def_levels and ref_levels separately, how can we derive number
of nulls in each level precisely?
> [Format] Add statistics that reflect decoded size to metadata
> -------------------------------------------------------------
>
> Key: PARQUET-2261
> URL: https://issues.apache.org/jira/browse/PARQUET-2261
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-format
> Reporter: Micah Kornfield
> Assignee: Micah Kornfield
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
