emkornfield commented on code in PR #197: URL: https://github.com/apache/parquet-format/pull/197#discussion_r1301968222
########## src/main/thrift/parquet.thrift: ########## @@ -191,6 +191,62 @@ enum FieldRepetitionType { REPEATED = 2; } +/** + * Tracks a histogram of repetition and definition levels for either a page or column chunk. + * + * This is useful for: + * 1. Estimating the size of the data when materialized in memory + * 2. For filter push-down on nulls at various levels of nested structures and + * list lengths. + */ +struct RepetitionDefinitionLevelHistogram { + /** + * When present there is expected to be one element corresponding to each repetition (i.e. size=max repetition_level+1) + * where each element represents the number of time the repetition level was observed in the data. + * + * This value should not be written if max_repetition_level is 0. + **/ + 1: optional list<i64> repetition_level_histogram; + /** + * Same as repetition_level_histogram except for definition levels. + * + * This value should not be written when max_definition_level is 0. + **/ + 2: optional list<i64> definition_level_histogram; + } + +/** + * A structure for capturing metadata for estimating the unencoded, uncompressed size + * of data written. This is useful for readers to estimate how much memory is needed + * to reconstruct data in their memory model and for fine grained filter pushdown on nested + * structures (the histogram contained in this structure can help determine the + * number of nulls at a particular nesting level). + * + * Writers should populate all fields in this struct except for the exceptions listed per field. + */ +struct SizeEstimationStatistics { Review Comment: Values are exact. would SizeStatistics be more meaningful from you perspective? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org