rich7420 commented on code in PR #6125:
URL: https://github.com/apache/datafusion-comet/pull/6125#discussion_r4405285168
##########
native/core/src/parquet/eager_page_index_reader_factory.rs:
##########
@@ -240,6 +277,91 @@ struct EagerPageIndexReader {
metadata_cache: Arc<FileMetadataCache>,
metadata_size_hint: Option<usize>,
spark_variant_schema: bool,
+ /// Empty handle whose registration every fetched buffer's reservation
shares.
+ reservation: Option<MemoryReservation>,
+ /// Holds the decode-buffer estimate while this file is being read.
+ _decode_reservation: Option<MemoryReservation>,
+}
+
+/// Data page size limit that both parquet-mr and parquet-rs default to.
+const DEFAULT_PAGE_SIZE: usize = 1024 * 1024;
+
+/// Estimates what a scan holds while decoding beyond the compressed column
chunks, which
+/// [`ReservedBytes`] charges exactly. arrow-rs allocates the rest internally
with no hook to
+/// observe it: per projected leaf column, a decompressed data page bounded by
the writer's page
+/// size, plus the output batch being assembled. Dictionary pages are left
out: they are often
+/// absent or small, and counting them at the same bound doubled the estimate
against TPC-H
+/// measurements. Files written with larger pages or large dictionaries are
underestimated.
+pub(crate) fn decode_buffer_estimate(projected: &Schema, batch_size: usize) ->
usize {
Review Comment:
A 500-column, one-row file reserves roughly 508 MiB before metadata loads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
